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Executive Summary 


The Louisiana Striving Readers evaluation assessed the implementation and effectiveness of the 
Voyager Passport Reading Journeys (PRJ), a widely used supplemental literacy intervention for 
struggling adolescent readers that reflects the research-based practices recommended by the 
National Reading Panel (2000) and other more recent syntheses (Biancarosa & Snow, 2004; 
Edmonds, et al., 2009; Kamil, et al., 2008; Scammacca et al., 2007; Torgesen et al., 2007). To 
date, PRJ has been adopted in 45 states across the country in almost 470 districts and over 2,200 
schools, and has served over 268,000 students. PRJ offers four levels of instruction appropriate 
for middle and high school students. The PRJ curriculum uses direct, explicit instruction in 
reading comprehension, vocabulary, and word study for adolescents who struggle with reading 
using age-appropriate fiction and non-fiction texts. The program is delivered through 50-minute 
daily lessons delivered 5 days a week. Assessments are embedded in the curriculum to enable 
teachers to monitor progress and differentiate instruction. The program is formatted as a series of 
15 two-week reading expeditions focused on topics related to science or social studies with 
optional add-ons (reteach or writing). Each week, students spend four days on lessons designed 
to build their fluency, vocabulary, and comprehension. They spend the fifth day online using 
SOLO®, an interactive online learning package. A library of Lexile-leveled books and magazines 
on age-appropriate topics is also provided for each classroom. 


The Louisiana Striving Readers Program, funded by the Louisiana Department of Education 
(LDOE) through a grant from the U.S. Department of Education, targeted over 1,200 struggling 
readers in grades 6-7 from ten middle schools across the state of Louisiana. The grant required a 
rigorous, independent experimental evaluation, conducted by SEDL, addressing fidelity of 
program implementation and program impacts on student motivation and reading achievement. 
Findings from previous quasi-experimental studies suggested that the program can be effective in 
improving the skills of adolescent readers, relative to other traditional approaches. Despite the 
program’s widespread use and promising research findings, PRJ had not been previously 
evaluated rigorously as part of an objective, large-scale, third-party study. 


The study reported here had two specific aims: 1) determine the fidelity of implementation, or 
the extent to which the program was delivered as the grant indicated it should be implemented; 
2) determine the impacts of PRJ on student reading and other related outcomes (i.e., student 
motivation and engagement in reading) and how the effects may have varied by student 
subgroups. The study design involved a multi-site randomized controlled trial in which students 
who were identified as reading Below Basic on the state reading assessment in 10 Title I middle 
schools in 4 districts in Louisiana were randomly assigned to one of two groups: 1) literacy 
intervention teachers who were trained in the PRJ program and delivered the curriculum as an 
add-on to students’ regular core reading curriculum, or 2) teachers who continued to deliver 
supplemental instruction as usual, which included a range of other services and electives 
available in their school (i.e., tutoring, study hall), which were not focused on supplemental 
literacy services or courses. The study followed teachers and students over one school year 
(2010-2011). 


The Louisiana state reading assessment ({LEAP) was used to identify eligible students in the fall 
prior to the start of the program and as a post-test outcome in the subsequent spring. For the 
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evaluation, SEDL also assessed students in reading achievement in the fall (pretest) and spring 
(posttest) of each year using a group-administered reading test (GRADE). Additional student 
outcomes, including student motivation and engagement in reading (MRQ), were also captured 
each fall and spring using student surveys. Fidelity of implementation was captured through 
classroom observations, archival school, program, and classroom data sources, and interviews 
with teachers, principals, PRJ coaches, and LDOE staff in the fall and spring of each year. 


Findings 


Implementation. The study assessed fidelity of implementation (FOI) — actual implementation 
relative to the ideal — in terms of the grant’s professional development (PD) and classroom 
implementation models. The grant involved a number of additional supports, particularly in the 
areas of professional development, than is “typical” with the PRJ implementation (e.g., LDOE 
staff site visits and TA, district coordinators support and TA, principal classroom visits, number 
and length of Voyager coaches follow up visits, and teacher credentials for hiring). Perhaps 
because of this, the study found lower levels overall of FOI in terms of actual PD received, 
particularly in terms of on-going supports. Despite less than ideal PD FOI, classroom FOI was 
adequate across all teachers/classrooms in the fall and spring with the majority of teachers 
implementing with high levels of FOI to the PRJ classroom implementation model. Additional 
results indicate that relative to a more “typical” PRJ combined (professional development and 
classroom) implementation model, the program was implemented with medium to high levels of 
adequacy across the 10 schools in the study. 


Impacts. The study found evidence of a statistically significant treatment effect on students' 
reading performance, as measured by the overall GRADE reading assessment score, and for the 
vocabulary and reading comprehension subscales of the GRADE assessment. We found no 
evidence that treatment students’ and control students’ outcomes differed on the LEAP overall 
ELA score or Reading subscale. Finally, we found no evidence that the treatment impacted 
students' motivation to read, as measured by the MRQ total score or any of the four MRQ 
subscales. 


The study found statistically significant differential impacts of the PRJ program by student 
gender and grade for the overall GRADE reading and comprehension scores. The results suggest 
that for males and 7" graders, assignment to the PRJ treatment condition yielded positive and 
substantial gains in reading scores. There was a statistically significant positive differential 
impact also found for the iLEAP Reading subscore for 7" graders. No other statistically 
significant differential impacts were detected for the LEAP or MRQ outcomes. In addition, we 
found no differential impacts by student poverty status, ethnicity, or baseline proficiency in 
reading. 


The overall impact and differential subgroup impacts are encouraging. This study provides 
evidence that PRJ is effective at improving struggling 6" and 7" grade students’ reading 
performance on nationally norm referenced standardized assessments relative to other electives 
after only one year of the intervention. The size of the effects, particularly for overall reading 
(effect size = 0.27) and reading comprehension (effect size = 0.31) were statistically significant 
as well as educationally meaningful (PRJ student scores increased 3 and 4 NCE points, on 
average, respectively). The vocabulary effect size (effect size = 0.13) was statistically significant 
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but relatively small (PRJ student scores increased, on average, 2 NCE points). Males appeared to 
benefit more from the PRJ program (males scores increased 4-5 NCE points, on average), as did 
7th grade students (7th grade students in PRJ scores increased 2 NCE points, on average). These 
findings are particularly promising given the relatively small number of proven adolescent 
literacy programs for the struggling readers, who were the targets of this study. 


Implications for Research, Policy, and Practice 


This study adds a critical piece to the evidence-base for the PRJ program — a high-quality third- 
party experimental study assessing effectiveness. The results are encouraging and provide 
evidence that the program can impact reading outcomes on a standardized norm referenced 
assessment for students two or more grade levels behind their peers after one year of 
implementation. However, this study involved significant involvement of the Louisiana state 
education agency, which may represent a less than typical implementation of the program. We 
found no impacts on student 7_LEAP reading scores or on student motivation or engagement in 
reading. In addition, the funding for the study was ended after only one year of intervention. 
Thus, the extent to which these results may replicate in different contexts, for a broader range of 
reading outcomes, and over an extended period of time remains an important empirical question. 
The program, and the field, would likely benefit from further research addressing these 
questions. 


Many middle school students continue to struggle with reading proficiency. At least two-thirds 
of eighth grade students fail to reach proficient-level reading scores (National Center for 
Educational Statistics, 2011). Recent initiatives emphasize the critical role of early reading 
instruction in preventing reading difficulties and the need for effective interventions to bring 
students to proficient levels in reading and narrow the gap between them and their peers. With 
increased investment and expectations in reading programs to ameliorate reading difficulties in 
students who are significantly behind their peers, policymakers and school administrators need 
evidence regarding the utility of the PRJ program, and other similar programs, as a strategy for 
improving student reading achievement and addressing gaps in reading difficulties. This study 
provides evidence of a supplemental reading program’s impacts that may help to inform 
decision-makers — policy makers and practitioners — attempting to address this critical problem. 


The study findings also have potentially meaningful implications for communities of practice, 
particularly those working closely with adolescent students reading significantly behind their 
peers. The study found PRJ, which focuses on direct, explicit comprehension, vocabulary, and 
word study instructional strategies, was implemented with adequate levels of fidelity in 
classrooms in a relatively short period of time. In addition, PRJ had the largest impacts on 
reading comprehension outcomes for these students. These results may be particularly important 
given improved reading comprehension skills may also significantly help students in learning 
other content areas. 
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Introduction 


Significance of the Problem 


This study addresses the effectiveness of a widely used supplemental reading program for 
struggling adolescent readers that reflects the research-based practices recommended by the 
National Reading Panel (2000) and other more recent syntheses (Biancarosa & Snow, 2004; 
Edmonds, et al., 2009; Kamil, et al., 2008; Scammacca et al., 2007; Torgesen et al., 2007). This 
and other similar programs are increasingly used to address reading difficulties in middle and 
secondary school students where the demands on students have shifted from learning to read to 
reading to learn. Converging evidence from two decades of research suggests that with 
appropriate instruction, nearly all students can become competent readers (Denton & Mathes, 
2003; Lyon, Fletcher, Fuchs, & Chhabra, 2006; Mathes & Denton, 2002; Snow, Burns, & 
Griffin, 1998). Yet statistics indicate that approximately two-thirds of fourth and eighth grade 
students fail to reach proficient-level reading scores (National Center for Educational Statistics, 
2011). Recent initiatives emphasize the critical role of early reading instruction in preventing 
reading difficulties, recognizing that students who do not learn to read well by third grade are 
less likely to build vocabulary and interact with a wide variety of texts (Good, Simmons, & 
Kame’ enui, 2001). As a result, older students who continue to struggle to read often face 
difficulties in organizing and acquiring content knowledge from informational text (Snow and 
Biancarosa, 2003). Studies have found that reading ability is a key predictor of achievement in 
other content areas, such as mathematics and science (ACT, 2006). Such failure can have a long- 
term impact on students’ self-confidence, motivation to learn, performance in school, and 
success in life (Harris & Sipay, 1990; Juel, 1988; Stanovich, 1986, 2000), and reading 
difficulties are the most common reason for referral into special education (Donovan & Cross, 
2002). Despite these concerns, there is limited evidence that supplemental reading programs can 
impact reading outcomes for older students that are reading significantly below grade level. 


According to the Education Watch Report (The Education Trust, 2009), Louisiana is the only 
state in which the gap between African American and white students has narrowed significantly 
in both 4" grade reading and 8" grade math. Despite this success in reducing the achievement 
gap, over 26,500 students did not graduate from the state’s high schools in 2008. The lost 
lifetime earnings in Louisiana for that class of dropouts alone total nearly $6.9 billion (Alliance 
for Excellent Education, 2011). One of the most commonly cited reasons that students drop out 
of high school is that they do not have the literacy skills to keep up with the high school 
curriculum, which has become increasingly complex (Kamil, 2003; Snow and Biancarosa, 2003). 
In 2006, the state adopted the Louisiana Literacy Plan, a blueprint for improving students’ 
reading achievement. As the plan incorporates grade-specific instructional models, research- 
based programs and strategies, ongoing assessment, and job-embedded professional 
development, it served as a powerful springboard for the Louisiana Striving Readers Program. 
The program assessed the effectiveness of a research-based reading intervention targeted to 
struggling readers in 6" and 7" grades and provided evidence from a rigorous, independent 
experimental evaluation conducted by SEDL regarding program fidelity of implementation and 
impacts on student reading outcomes as well as outcomes related to student motivation and 
engagement in reading. 
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Overview of the Intervention 


The Louisiana Striving Readers Program used Passport Reading Journeys (PRJ), a 
comprehensive supplemental curriculum published by Voyager Expanded Learning, which 
blends targeted, teacher-led instruction with student-centered technology. To date, PRJ has been 
adopted in 45 states across the country in almost 470 districts and over 2,200 schools, and has 
served over 268,000 students. The program offers four levels of instruction appropriate for 
middle and high school students. PRJ uses direct, explicit instruction in comprehension, 
vocabulary, and word study for adolescents who struggle with reading using age-appropriate 
fiction and non-fiction texts. The program is delivered through 50-minute daily lessons delivered 
5 days a week. Assessments are embedded in the curriculum to enable teachers to monitor 
progress and differentiate instruction. The program is formatted as a series of 15 two-week 
reading expeditions focused on topics related to science or social studies with optional add-ons 
(reteach or writing). Each week, students spend four days on lessons designed to build their 
fluency, vocabulary, and comprehension. They spend the fifth day online using SOLO®, an 
interactive online learning package. A library of Lexile-leveled books and magazines on age- 
appropriate topics is also provided for each classroom. 


A limited body of research has examined the impact of the supplemental PRJ program on 
reading outcomes with middle and high school students. Shneyderman (2006) carried out a 
quasi-experimental study with 9" and 10" graders in Miami, Florida. The study involved four 
Voyager PRJ schools with predominantly low-achieving, Hispanic Limited English Proficient 
students (n=453) compared to four matched comparison control schools (n=394) identified using 
propensity score matching on pretest state assessment scores, student demographics, and English 
for Speakers of Other Languages levels. Results from Hierarchical Linear Modeling (HLM) 
analyses using the state assessment pretest as a covariate indicated statistically significant 
positive impacts on reading outcomes on the Florida Comprehensive Assessment Test for ninth 
graders (ES = +0.22) but not for tenth graders (ES = +0.12) with an overall effect size of +0.17 
(Slavin, Cheung, Groff, and Lake, 2008). 


Other smaller scale quasi-experimental evaluations of the Voyager PRJ program funded by the 
publisher suggest that, in comparison to other reading curricula, PRJ has been associated with 
better reading outcomes. Denson (2008) implemented PRJ with ninth graders who were 
struggling readers (based on previous years state assessment scores) in a high school in Dallas, 
Texas (n=88), and compared them to ninth grade students in a matched control school (n=82) 
based on pretest reading scores and demographic variables. The PRJ students made significantly 
greater progress on norm-referenced (ITBS: Iowa Test of Basic Skills) and criterion-referenced 
(TAKS: Texas Assessment of Knowledge and Skills) reading assessments relative to the students 
in the control school over two years. The study did not estimate effect sizes or establish that the 
students were statistically equivalent at pretest, which weakens the design and implications of the 
findings. In a more recent quasi-experimental study, Denson, Peyton, and Macpherson (2009), 
looked at ninth graders who were struggling readers from the same high school in Dallas, Texas 
(n=99) who participated in the PRJ program, and compared them to students matched on 
demographics and reading assessment variables from another high school in the district (n=46). 
Results from these analyses are less clear. The authors state that the PRJ students scored 47 
points higher on the TAKS although they do not indicate whether this difference was statistically 
significant. Again, the authors did not calculate the effect size for the difference and did not 
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confirm that there were no statistically significant differences between the treatment and control 
groups at pretest, thus weakening the validity of the results. 


The extent of evidence of the PRJ program is based on a small number of quasi-experimental 
studies. Only one independent quasi-experimental study has provided strong and direct support 
for the potential effectiveness of the program. Though the results from this and other quasi- 
experimental studies appear promising, the studies assessing the effectiveness of the PRJ 
program were conducted in a limited number of schools, involving predominantly statewide 
assessments of reading outcomes, and students in the early years of high school (i.e., 9" grade). 
Another key limitation across this research is the inability or failure to monitor and verify 
program implementation fidelity. The inability to examine dimensions related to implementation 
fidelity, dosage, and program duration or to document and verify differences between the control 
and treatment conditions weakens the conclusions that can be drawn. Thus, the extent to which 
these results may replicate in different contexts and for a broader range of reading outcomes 
remains an important empirical question. The results provide some evidence that the PRJ 
program can be effective. But perhaps more importantly, this evidence suggests that, in 
combination with the widespread adoption of the program, the program is well positioned for an 
independent efficacy trial to measure implementation fidelity and determine its impact on 
reading outcomes. 


Findings from quasi-experimental studies suggest that the program can be effective in improving 
the skills of adolescent readers, relative to other traditional approaches. Yet, despite the 
program’s widespread use and promising research findings, PRJ has not been evaluated 
rigorously on a scale as part of an objective, third-party efficacy study. 


A number of factors make information from an efficacy study of the PRJ program valuable for 
educational policymakers and practitioners, particularly at the state and local levels: 


¢ In response to NCLB, schools’ adoption of core reading programs has steadily increased 
nationwide. However, many students continue to struggle with reading proficiency, 
particularly in late elementary, middle, and high school, and need intense and effective 
interventions to bring them to proficient levels in reading and narrow the gap between them 
and their peers in reading, and in other content areas impacted by reading. PRJ is among the 
most widely used supplemental reading programs with years of delivering methods found to 
be effective in reading instruction for adolescents. 


¢ Results from a quasi-experimental study suggest that PRJ can produce student-level effects 
of statistical and educational significance on reading outcomes. However, this study was 
conducted in select schools in a single district and offer only preliminary indications of 
program effects. A third-party efficacy study can test whether the PRJ program has similar 
results across a larger sample of districts and schools. 


e With increased investment and expectations in supplemental reading programs to ameliorate 
reading difficulties in students who are significantly behind their peers each year, 
policymakers and school administrators need evidence regarding the utility of the PRJ 
program as a long-term strategy for improving student reading achievement and preventing 
gaps in reading difficulties. 
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Overview of the Evaluation 


The study reported here had two specific aims: 1) determine the extent to which the program was 
delivered as the grant indicated it should be implemented or fidelity of implementation; 2) 
determine whether the program produced substantial impacts on student reading and other 
related outcomes (i.e., student motivation and engagement in reading) and how the effects of the 
program may have varied by student subgroups. The study design involved a multi-site 
randomized controlled trial in which students who were identified as reading Below Basic on the 
state reading assessment in 10 Title I middle schools in 4 districts in Louisiana were randomly 
assigned to one of two groups: 1) literacy intervention teachers who were trained in the PRJ 
program and delivered the curriculum as an add-on to their regular core reading curriculum, or 2) 
teachers who continued to deliver supplemental instruction as usual, which included a range of 
other services and electives available in their school (e.g., tutoring, study hall) which were not 
focused on supplemental literacy services or courses. The study followed teachers and students 
over one school year (2010-2011). 


Research Questions 
The study addressed the following major research questions: 


Research Question 1: Fidelity of Implementation (FOI). Describe the extent to which the 
intervention was delivered as the grant indicated it should be implemented. Was there significant 
variation in implementation fidelity among the schools and classrooms in the treatment group? 


Research Question 2: Overall Impacts on Student Reading Outcomes. Does student-level 
assignment to the PRJ supplemental literacy program produce impacts on reading achievement 
for 6" and 7" grade students significantly behind their peers (i.e., scoring Below Basic on the 
state assessment) in reading relative to student assignment to a supplemental non-reading 
elective services control condition? 


Research Question 3: Overall Impacts on Student Motivation to Read. Does student-level 
assignment to the PRJ supplemental literacy program produce impacts on students’ motivation to 
read for 6" and 7" grade students significantly behind their peers (i.e., scoring Below Basic on 
the state assessment) in reading relative to student assignment to a supplemental non-reading 
elective services control condition? 


Research Question 4: Overall Impacts by Subgroups. For which subgroups and under which 
conditions does the program have its greatest impact? 


Study Design and Methods 


SEDL’s approach for assessing the implementation and impact of the PRJ program involved two 
key elements: 


¢ Multi-site Randomized Controlled Trial (RCT) in which eligible 6" and 7" grade students 
who scored Below Basic on the Louisiana state assessment in reading in the spring 2010 from 
10 middle schools were randomly assigned to one of two groups: 1) literacy intervention 
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teachers that were trained in the PRJ program and deliver the curriculum, and 2) teachers that 
delivered supplemental or elective instruction not focused on literacy services. Statistical 
power calculations indicated that 10 schools would be needed to detect an effect size of .10 
for main effect and subgroup analyses (n=1,200 students) so the study design targeted a 
sample of approximately 1,200 students in 10 schools across 4 districts in Louisiana. The 
study followed teachers and students through one school year (2010-2011). 


Implementation Study to document the specific fidelity of implementation of the 
supplemental intervention program in each classroom and school as well as the level of 
implementation or “dose” (i.e., intensity, duration, and integration of services) of 
professional development services to determine if the supplemental intervention was 
delivered as intended, and to collect data about why, how, and under which conditions it was 
or wasn’t implemented with fidelity. The implementation study also documented general 
fidelity constructs related to implementation (i.e., teacher and classroom factors associated 
with high fidelity of implementation) or other facilitators and barriers influencing the 
program implementation which may be significant predictors of program outcomes in 
treatment classrooms and may explain indirect impacts as well as the critical contextual 
factors related to the program effects. 


Sampling Plan 


A multi-stage sampling process was refined and implemented during the first year planning 
period of the study, involving: 


Initial school selection. The LDOE recruited a purposive sample of schools that met 
eligibility criteria for inclusion in the sample (i.e., demonstrated the required number of 
potential struggling readers for each grade proposed for the study, met Title I eligibility 
criteria, and secured letters of agreement to participate in the study from district 
superintendents and school principals). 


School eligibility. To be included in the study, schools had to have a minimum number of 
eligible striving readers to create treatment and control groups at each grade level. Eligibility 
was determined from information derived from LDOE school databases. Initial estimates 
from the state indicated that the 10 schools willing to participate in the study met the 
eligibility criteria. 


Student sample selection. Based on recruitment efforts and projections of student 
achievement from the state assessment (Integrated Louisiana Educational Assessment 
Program or iLEAP) in each school, a sample of 10 schools and an initial pool of 
approximately 1,400 eligible students (i.e., scoring Below Basic in reading) were targeted. 
SEDL researchers confirmed these data during the first year planning period to verify school 
eligibility and the initial sample. Final requirements and memoranda of understanding were 
negotiated with each school during the first year planning period. Critical to this agreement 
was the schools’ commitment to change their enrollment procedures to involve random 
assignment for all consenting students and the participation of staff in the study. 
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¢ Random assignment. At each school, enrollment procedures began in the summer 2010 by 
determining a pool of eligible struggling readers who scored Below Basic using the previous 
spring iLEAP criterion-reference scores for reading. From this pool of students, school 
liaisons (program staff trained and supported by the study team) provided information about 
the study to parents and solicited their consent for student participation in the study. The 
SEDL study team (i.e., data manager) randomly assigned half of the eligible, consented 
students to the PRJ program and half to the control condition. Students who were tested in 
the spring 2010 but did not return or show up for school in the fall 2010 were considered 
ineligible for the study. 


Key Measures of Outcomes 


Louisiana state reading assessment ({LEAP) data was used to randomize students (spring 2010) 
as well as a post-test outcome the subsequent spring. Students were also assessed in reading 
achievement in the fall (pretest) and spring (posttest) using a group-administered reading test 
(GRADE). Additional student outcomes, including student motivation and engagement in 
reading (MRQ) were captured in the fall and spring using surveys. Fidelity of implementation 
was captured by classroom observations and interviews with teachers, principals, PRJ coaches, 
and LDOE staff in the fall and spring of each year as well as archival school, program, and 
classroom data sources. 


Overview of the Report 


The remainder of the report is divided into three parts. Part I is an overview and in-depth 
description of the intervention, part II describes the implementation study design and results, and 
part III presents the impact study design and results. Appendices include additional details about 
the analytic methods and measures used in the study. 
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Description of the Intervention 


Description of the Study Sample 


Selection of Study Schools. A purposive sample of 10 middle schools were recruited that met 
eligibility criteria for inclusion in the sample. Schools had to enroll the required number of 
potential struggling readers for each grade (i.e., 6" and 7” grades) proposed for the study and had 
to sign letters of agreement to participate in the study. School and student demographics had to 
meet Title I eligibility criteria. Eligibility was determined from information derived from LDOE 
school databases. State data indicated that the 10 schools willing to participate in the study met 
the eligibility criteria. Table 1.1 below displays sample pool estimates based on spring 2010 
iLEAP data provided by LDOE for all 10 schools indicating commitments to participate in the 
study. 


Table 1.1. Expected Student Sample Pool Based on 2010 iLEAP State Assessment Data 


6th Grade Students 7th Grade Students 

n Below Basic/ | % Below | % African- | n Below Basic/ | % Below | % African- 

School | Locale Total Basic American Total Basic American 
1 Rural 87/185 47% 36% 94 / 204 46% 35% 
2 City 64 / 163 39% 91% 66 / 148 45% 91% 
3 City 55/134 41% 93% 76/150 51% 93% 
4 City 64 / 146 44% 97% 50/96 52% 98% 
5 City 48 / 150 32% 90% 57/ 138 41% 83% 
6 Suburb 109/211 52% 61% 67 / 164 41% 63% 
7 Rural 82/198 41% 22% 59/156 38% 31% 
8 City 53/147 36% 74% 50/126 40% 71% 
9 Rural 39 / 84 46% 73% 37/76 49% 55% 
10 Rural 56/ 138 41% 41% 64 / 138 46% 39% 


Final requirements and memoranda of understanding were negotiated with each school by the 
LDOE, and sites were developed and prepared for the study through meetings during the first 
planning year of the grant. As part of this process, SEDL researchers presented a detailed 
description of the RCT study procedures and considerations for reducing any disruptions to 
normal school procedures. Critical to this period of site development were site visits to each 
school to finalize the parameters and ideal conditions for the implementation of the research in 
each of the schools and to provide an opportunity to answer any questions administrators may 
have had before the study began. 


Selection of Study Intervention Teachers. PRJ was implemented by literacy intervention 
teachers that were hired by the school districts based on selection criteria and requirements 
established by LDOE' to provide the supplemental program activities during an elective period 


' LDOE criteria and requirements, include: 1) Valid Louisiana teaching certificate; meeting NCLB definition of 
Highly Qualified Advanced degree in reading or reading specialist certification (preferred); 2) Knowledge of 
reading intervention and scientifically-based research instructional strategies; 3) Knowledge of specific strategies 
used before, during, and after reading to build comprehension; 4) Knowledge of strategies and activities to increase 
vocabulary and oral language development; 5) Ability to communicate effectively with students, school personnel, 
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devoted to this offering at each school. LDOE worked closely with districts to staff literacy 
intervention teachers who taught exclusively in the intervention classrooms rather than in 
addition to their duties in regular ELA classrooms and to have one intervention teacher per 
school. Special care was also taken to ensure that treatment teachers did not interact with 
students assigned to the control condition. Based on estimates of eligible students, each school 
planned to have, a range of 2-6 sections with up to 20 students in each section, participating in 
PRJ. 


Selection of Study Participants (Students). The study focused on 6" and 7" grade students 
scoring Below Basic on the spring 2010 Integrated Louisiana Educational Assessment Program 
({LEAP) English Language Arts (ELA) subscale. The iLEAP is Louisiana’s state assessment and 
contains items linked to established Grade Level Expectations (GLEs) criterion referenced items 
and items from the Iowa Test of Basic Skills ITBS) norm referenced items. Students who score 
Below Basic are considered to be performing below grade level and are often targeted for 
interventions. The iLEAP was administered during March of the previous spring, prior to 
entering 6" and 7" grade. While the iLEAP tests students’ acquired skills across several key 
areas (1.e., science, math and ELA), only the ELA score was used to establish eligibility for 
inclusion. The researchers worked with the LDOE staff to obtain spring 2010 iLEAP scores for 
all 5" and 6" grade students across the state. The data were provided in late May and researchers 
began working with participating districts and schools to identify their incoming 6" and 7” 
graders. Students were excluded from participation if they did not have a spring 2010 LEAP 
ELA score. 


The total number of students in the 6" and 7" grade, was 3,595 across the 10 study schools. 
From this pool, 1,437 were eligible for randomization, based on their spring 2010 iLEAP ELA 
score in the Below Basic (Approaching Basic and Unsatisfactory) category. The remaining 2,158 
were not eligible for randomization. All students with a spring 2010 iLEAP score were eligible 
to participate, including limited English proficient (LEP), and special education students. 


Logic Model for the Intervention 


SEDL researchers created a logic model (see Figure 1.1 below) to describe the linkages between 
PRJ professional development, curriculum materials, instruction, and proposed short and long- 
term outcomes. The model provided a conceptual framework for the evaluation approach, the 
research design, core constructs to be measured, and the timeline for the study. It hypothesized 
the major constructs that are relevant for the evaluation of the PRJ program and the pathways 
through which students’ reading achievement may be influenced. 


Key components of PRJ (listed under Program Inputs/Activities) included teacher, principal, and 
district coordinator professional development; teacher and student curricula and materials; and 
ongoing support throughout the school year. As the logic model indicates, the PRJ curriculum 
was hypothesized to be mediated by how well teachers implement the program (see Short Term 


and parents, and an ability to work collaboratively with others; 6) Ability to promote student motivation and 
engagement in learning; 7) Knowledge of technology and how it is used to facilitate instruction and track student 
progress in reading; and 8) a minimum of three years of demonstrated effective classroom instruction, including 
teaching reading or core reading program and experience in improving student performance in diverse populations. 
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Teacher Outcomes). Similar to many other programs, a critical component of PRJ was the way in 
which teachers combine specific instructional practices and curriculum materials in the 
classroom. For example, the fidelity with which teachers implement the PRJ materials and 
methods would potentially mediate short-term outcomes, including changes in teacher 
instructional practices and facilitation of student motivation and engagement in reading. Long- 
term outcomes include teacher efficacy with the PRJ materials and strategies, and students’ 
performance in reading. 
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Logic Model for Passport Reading Journeys - Targeted Intervention for Louisiana Striving Readers 


PROGRAM INPUTS AND ACTIVITIES INSTRUCTIONAL PRACTICES SHORT TERM OUTCOMES LONG TERM OUTCOMES 


Professional Development 


Launch Training 

Face to face, 2-day group training covering assessment measures, 
student grouping, classroom setup, curriculum materials, 
implementation models, classroom management, lesson delivery 
practice, and use of VPORT 


Cohort Meeting 
All participating staff meet to network, share successes and challenges, 
review data, and receive updates/information on program activities 


Online Product Training 

Overviews of curriculum, assessment, and implementation in an 
interactive, self-paced environment available 24/7 

Principal Training 

Initial training participation encouraged; Principal's Handbook provided 


to guide support of teachers, program implementation (e.g., walk- 
throughs, scheduling, technology) 


Ongoing Support 


Online support for teachers providing interactive training and assistance 
in use of VPORT and other self-selected topics 


Principals improve quality of implementation through structural 
(e.g., scheduling, resources) and procedural (e.g., walk-throughs, 
meetings) supports 


Ongoing consultative support provided by implementation specialists in 
the form of lesson modeling, classroom observation and feedback, side- 
by-side coaching, and data review 


Additional support provided by LDOE staff online and during regularly 
scheduled site visits 


Program Materials 


Teacher Materials 
Teacher Edition of Book, Assessment Guide, Word Study Guide, Library 
Guide, Writing Extensions Guide, and Expedition DVDs 


Student Materials 
Student Book, Word Study Book, Assessment Book, Anthology, and 
SOLO Access 


Passport Reading Journeys Library 
VPORT Data Management System 
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6th and 7th grade students reading below grade 
level expectations using 10-day Expeditions, 
: each with a 1-day reteach add-on. The i 
: Expedition thematic units support best practices: 
: in adolescent reading instruction : 
: Teachers present lessons that provide explicit, — : 
systematic instruction in critical reading skills, 
including: vocabulary, comprehension, text 


meaning and interpretation, decoding, fluency, 
and writing 


Teachers arrange classroom to accommodate 
whole group and small group instruction as well 
as independent practice 


Teachers monitor student skill acquisition (e.g., 
comprehension and vocabulary) via formative 
assessments at the end of each Expedition, as 

well as through assessments in SOLO technology: 
component : 


Students use high-interest texts, DVDs, and 
web-based technology tool (SOLO) 


Additional reading practice for students provided ==: 
by library of Lexile-leveled books and magazines on : 
age-appropriate topics provided in each classroom: 


Teachers make progress toward 
high fidelity of implementation with 
instructional materials 


Teachers improve in use of instructional Teacher efficacy with use of intervention 


practices related to reading ee 
assessment and monitoring tools 


program materials, technologies, strategies, 


Teachers improve facility with data- 
driven instruction, monitoring student 
progress, and tailoring individual 
instruction effectively 


Improved student achievement in reading 


Students show increased use of 
strategies for reading comprehension, 
building vocabulary, and improving 


fluency and writing 


Students show increased motivation 
and engagement in reading 
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Plan for Implementing the Intervention 
Planned Professional Development Model 


The PRJ curriculum and materials are important elements of the project, but it is the teacher’s 
ability to implement the program with fidelity that results in a long-term difference for learning. 
Therefore, the professional development for teachers was intended to be intensive and ongoing, 
offering a full range of support that included lesson modeling, side-by-side coaching with 
observation and feedback, and data review. The professional development model, delivered 
and/or facilitated by LDOE staff and Voyager implementation specialists, included five formats 
for literacy intervention teachers and project support staff: Launch Training, Online Product 
Training, Ongoing Consultative Support, Cohort Meetings, and Principal Training. 


In addition to the professional development provided by Voyager, the literacy intervention 
teachers received technical assistance from the LDOE and their principals. The technical 
assistance from LDOE was provided via on-site visits, phone conversations, and emails. All 
technical assistance was documented by LDOE through a T/TA tracking system. LDOE staff 
planned to visit each school at least eight times each year to provide technical assistance during 
and in between Voyager implementation specialists visits to ensure program fidelity. District 
coordinators would provide additional support by coordinating project efforts at the local level 
and serving as liaisons between and among partners. Principals planned to visit classrooms and 
work with teachers to ensure fidelity of implementation is maintained across the project’s 
timeline. Principals planned to visit each teacher once a week during the initial month of 
implementation and then twice a month over the school year. In total, the project planned to 
provide a minimum of 86 hours of professional development to teachers over the school year. 
District and LDOE staff would receive a total of 30 and 18 hours, respectively, and Principals 36 
hours across the planning year and first school year of implementation. These supports described 
in detail in this section are outlined in Table 1.2 below. 


Table 1.2. Professional Development Model: Method, Frequency, Purpose, and Contact Hours” 


YEAR 1 
2009-2010 
(PLANNING YEAR) 


Method/Format Frequency Purpose 
¢ Preview PRJ curriculum 
Cohort Meeting Spring 2010 * Discuss program activities 
¢ Attended by District staff & Principals 


° LDOE has enhanced Voyager’s PD model by providing additional supports highlighted in yellow, for each year of 
the study. 
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YEAR 2 
2010-2011 
(FIRST YEAR OF IMPLEMENTATION) 

Method/Format Frequency Purpose 
Launch Training August 2010 Face-to-face PRJ implementation training covering: 
Assessment measures 
Student grouping 
Classroom setup 
Curriculum materials 
Implementation models 
Classroom management 
Lesson delivery practice 
Online Product Ongoing Self-paced PRJ modules that supplement Launch 
Training Training. Topics include: 
Assessment and Data Management 
Classroom Management 
Lesson Preparation 
Model Lessons (video) 
Trouble Shooting Tips 
Ongoing 2x in September On-site school visits provided by Voyager 
Consultative 1x in Oct. — Jan. implementation specialists to support skill 
Support 1x in March — May | development and program implementation. 
Activities include: 
2 to 4 hours per ¢ Classroom observations and feedback 
visit Lesson modeling 
Side-by-side coaching 
Data analysis and decision-making 
Exit conference with principal 
Completing Fidelity of Implementation 
Checklist 
Cohort Meeting February 2011 All participating staff in project schools meet to: 
¢ Network 
¢ Share successes and resolve challenges 
¢ Review data 
* Receive program updates 
Principal Meeting | Fall 2010 In addition to attending Launch Training and 
Cohort Meetings, all principals in project schools 
meet to: 
¢ Define the principal’s role in program 

implementation (e.g. walk-throughs, 

scheduling, technology) 

Review the Principal’s Handbook for PRJ that 

contains tools used to guide support of teachers 
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YEAR 2 
2010-2011 
(FIRST YEAR OF IMPLEMENTATION) 
Method/Format Frequency Purpose 


; E Staff Jn 1 } \ al [ all p et ticipan a 
| needed 
POC Tra Furs (Teachers) | 86] 


Teacher Training and Support from Voyager. Initial group or “launch” training with 
intervention teachers, school principals, and district coordinators was planned over two days in 
early August 2010 with the intent of covering the key features of the PRJ program, including: 
assessment measures, student grouping, classroom set-up, curriculum materials, implementation 
models, classroom management, and lesson delivery practice. Launch training was delivered by 
Voyager implementation specialists. SOLO® (Strategic Online Learning Opportunities) was part 
of the PRJ curriculum, which provides opportunities for students to practice skills learned during 
face-to-face instruction. VPORT® (Voyager’s online data management system), imports 
assessment data, monitors progress, and generates reports. On VPORT”, under the Training tab, 
teachers accessed the Online Product Training (OPT) modules. These were interactive, self- 
paced tutorials, which provided professional development targeted to the PRJ curriculum. 
Teachers were required to log 8 hours of PRJ online training via the VPORT® system and 
complete 4 modules (i.e., Introduction, Curriculum Overview, Assessment Overview, and 
Implementation Overview). Upon completion of each module, the teacher received a certificate 
of completion. In addition, teachers participated in an additional 8 hours of online training, 
which was self-selected from a menu of available online support materials including video 
presentations, demonstrating before, during, and after reading strategies, advanced word study, 
reading, and classroom management. 


Ongoing consultative support occurred monthly and provided in-class coaching that included 
lesson modeling, classroom observation and feedback, and data review. Voyager implementation 
specialists provided services that corresponded to the needs of each teacher. In the second year of 
the grant, 2010-11, Voyager implementation specialists planned to visit each teacher involved in 
the SR grant a total of nine times: two visits in September; and one visit in each month from 
October to May, except for February (when the Cohort Training would serve as Voyager 
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"support"). Each visit to a teacher was to range in time from a minimum of two to a maximum of 
four hours. 


Teacher Supports from School, District, and State Staff. Voyager recommended principals 
visit PRJ classrooms on at least two occasions during each school year, once during the initial 
implementation phase (September or October) to verify that PRJ instruction has begun, and once 
during the ongoing implementation phase (November through the end of the school year) to 
determine the degree to which teachers have implemented the program. The Louisiana 
Department of Education (LDOE) planned to implement this portion of the Voyager Professional 
Development model with additional enhancements. During the initial months of implementation 
(i.e., August and September 2010), the principal would visit the classroom weekly for an entire 
class period to ensure PRJ instruction began and to address start-up needs. During subsequent 
months, the principal would visit a class period every two weeks to determine to what degree all 
components of the program were being utilized (e.g., classroom library, SOLO®, appropriate 
pacing, etc.). The principal would summarize the data from observations and student data reports 
to provide feedback to the teacher on the level of implementation observed using the tools 
provided by PRJ. 


Planned supports would also be provided by district coordinators who were designated by the 
grant to facilitate implementation in the schools. District staff would attend PRJ trainings and 
provide a range of supports to teachers implementing the curriculum in district schools, from 
logistical supports to direct instructional assistance. Project staff from the Louisiana Department 
of Education would also visit each school at least eight times a year to provide technical 
assistance during and in between Voyager implementation specialist visits to ensure program 
fidelity. During their visits, the LDOE staff would observe the intervention classroom and review 
student data. They would meet with the principal and the intervention teacher to review 
observations regarding program implementation. 


Project Meetings Involving All Sites and Staff. In Spring 2010 and 2011, cohort meetings 
would be conducted in order to bring all teachers and staff associated with the SR grant together 
to network, share successes and challenges, review data, and receive updates and information on 
program activities. During the first year of the project, the first cohort meeting would be attended 
by district coordinators, principals, LDOE staff, SEDL evaluators, and Voyager PRJ 
representatives. The purpose of the first cohort meeting would be to provide information on the 
Voyager PRJ program and first-year program activities, including trainings, site visits, timelines, 
record keeping, etc. In the second year of the project, the cohort meeting was designed to provide 
updates and to give schools opportunities to network with each other. Additionally, a principal 
meeting would be held in year 2 (fall 2010) of the grant. The purpose of this meeting would be to 
explain the principal’s role in program implementation (e.g., walkthroughs, scheduling, 
technology needs). 


Planned Classroom Model 
PRJ was designed to provide four (4) distinct academic years worth of age-appropriate and 
targeted reading intervention. Students who need and receive more than one year of intervention 


will extend their skills with new content. Two levels of PRJ were involved in the SR Grant in 
LA, PRJ Beginnings (for 6" graders) and PRJ I (for 7" graders). Each level of PRJ was designed 
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for one full academic year with 15 two-week Expeditions with optional add-ons (reteach or 
writing). While both PRJ Beginnings and PRJ I teach similar strategies and reinforce common 
skills, there are unique topics and Expedition content that are age/grade appropriate and of 
increasing complexity. Participation in PRJ I does not require completion of PRJ Beginnings. 
The planned classroom model, when fully implemented, involved core instructional elements in 
reading, reading materials, formative assessments, and use of technology, each described in more 
detail below. 


Core Instructional Elements in Reading 


Instructional Practices. For the Striving Readers grant, PRJ was planned to be implemented by a 
literacy intervention teacher during the regular school day in small classes (e.g., 15 students) and 
in addition to students’ reading and English language arts coursework. Released in 2005, PRJ 
aligns fully with the state’s ELA Content Standards and complements the core reading programs 
(Holt McDougal and Harcourt) used by the districts that participated in the project. PRJ followed 
a standard protocol that provides daily, 50 minute lessons that provide explicit, systematic 
instruction in critical reading skills. The curriculum incorporated fast-paced video segments on 
DVD, engaging text, and online interactive lessons. It was formatted as a series of 15 two-week 
reading Expeditions with optional add-ons (reteach or writing) on engaging topics related to 
science, social studies, and literature. A library of Lexile-leveled books and magazines on age- 
appropriate topics was provided for each classroom. 


PRJ organized instruction around two, two-day routines to facilitate grouping and differentiation. 
The first and third days’ lessons are whole-group instruction in which students are introduced to 
new vocabulary and a new reading passage. The second and fourth days’ lessons include whole- 
group review of the previous day’s instruction and the opportunity for students to independently 
(or with a partner) re-read the passage to build fluency. Independent and small-group structured 
practice allows the teacher to work intensively with some of the students. The fifth day is spent 
in independent or paired practice on SOLO® (Strategic Online Learning Opportunities), the 
technology component of PRJ. See Table 1.3 for a sample lesson cycle illustrating a composite 
summary of all 10 lessons in an Expedition. Lessons 1, 3,6, and 8 follow the same pattern, and 
Lessons 2, 4, and 7 follow the same pattern. Lessons 5 and 10 are for online instruction. Lesson 9 
is unique in that it contains the formative assessment and closing video segment for the 
Expedition, in addition to the Lesson 8 review. After administering the formative assessment 
within Lesson 9 and reviewing student scores, teachers may decide to reteach and reinforce the 
skills introduced in the Expedition. The reteach lesson, is taught immediately after Lesson 10 
(SOLO) in each expedition. Each reteach highlights skills related to either comprehension and/or 
vocabulary and includes hands-on practice activities, as well as an additional assessment tool. 
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Table 1.3. Sample Lesson Cycle Passport Reading Journeys (PRJ) 10-Day Lesson Cycle with 1-day Reteach Lesson 


WHOLE Group WHOLE Group WHOLE Group 
Advanced Word Before Reading Before Reading, 

Study (10 min) (10 min) Reading (20 min) 

°  Affixes * Vocabulary * Vocabulary Review 


Sight Words Review * Reread Two-Day 

Multisyllabic Applied Lesson Passage 

Words Vocabulary 

Spelling 

Fluency 
Before Reading Reading Assessment The Reteach Lesson 
(15 minutes) (15 minutes) (25 minutes) is taught following 
* Expedition ¢ Reread Two-Day * Tips for Success Lesson 10 (SOLO) 

Introduction Lesson Passage Gumep ¢* Comprehension and for each Expedition 


(Lesson I only) Practice aa Vocabulary based on the reteach 


* Expedition Opener Comprehension wascwein Assessment (on a guidelines found in 
DVD Strategies new reading Lesson 9. 
oo =} SOLO 
J (Lessons I and 6 | (50 minutes) passage) 
x only) z : * Practice with High- Comprehension and 
, a | Guided . ; 
=| °* Vocabulary wm | - Stakes Testing vocabulary skills are 
<|°* Comprehension 2 —— Format reinforced that have 
Strategy; application of ‘ cary 
= fevered eS 


Reading (20 minutes) INDEPENDENT SMALL | TEACHER-DIRECTED strategies 
* Read Two-Day GROUP SMALL GROUP* 
Lesson Passage After Reading Word Study 
Model Vocabulary (10 minutes) (25 minutes) 
and ¢ Comprehension ¢ Word Recognition 
Comprehension Check * Vocabulary 
Strategies Writing in ¢ Fluency Reading 
Response to 


students. 


Reading dus 
After Reading PAIRED FLUENCY Resear toe Expedition Wrap-Up 


(5 minutes) PRACTICE cline katie (5 minutes) 
* Review PRJ Library pie * DVD Closing 


Vocabulary and (15 minutes) A Segment 
Comprehension * Building Fluency raf nla dy = . eae Probing 
Strategies * Self-Selected ‘Sica te ancl Questions 
Texts directed Word Study. 
This page illustrates a composite summary of all 10 lessons in an Expedition. 
Lessons 1, 3, 6, and 8 follow the same pattern. Lessons 2, 4, and 7 follow the same pattern. Lessons 5 and 10 follow the same pattern. 
Lesson 9 is unique in that it contains the formative assessment and closing video segment for the Expedition, in addition to the Lesson 8 review. 
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Explicit Vocabulary Instruction. PRJ addresses vocabulary development using explicit 
instruction of word meanings (Marzano, 2004; Schatschneider et al., 2004), work context, and 
development of strategies to determine unknown words (Bauman et al., 2005). Affixes and roots 
are taught to students in a sequential pattern that is supported by the identified words in passages. 
A carefully planned sequence of vocabulary skills and word choices (e.g., high-utility words) is 
meshed within the student passage reading and comprehension activities (Baker, Simmons, & 
Kame’ enui, 2004; Beck, McKeown, & Kucan, 2002). The sequence provides an introduction that 
is supported by numerous repetitions of the new words through activities such as discussion, 
writing, and extended reading. Lessons promote word consciousness and knowledge of word 
categories, such as antonyms and synonyms, to help student make connections between words. 


PRJ’s interactive web-based reading resource, SOLO”, provides self-paced, bi-modal (audio and 
visual stimuli) reading practice to build vocabulary. Multiple tools help students determine word 
meaning and contextual use in self-selected, Lexile-leveled reading passages. New words are 
introduced with student-friendly definitions and examples. Supports include automated clues or 
prompts and a function that allows students to click on difficult words to hear their pronunciation 
and definition. Students create their own word banks by “depositing” two words of their 
choosing from the previous SOLO® reading passage. As the word banks grow, repeated 
exposures help students to learn different meanings of the new words and how to use them in 
multiple contexts. 


Direct and Explicit Comprehension Strategy Instruction. PRJ integrates comprehension 
instruction with vocabulary instruction and provides strategies to help students make sense of 
text. These strategies include, but are not limited to, summarizing, asking and answering 
questions, paraphrasing, predicting, making inferences, and finding the main idea. PRJ provides 
direct and explicit instruction for teaching students how to use comprehension strategies. As the 
lesson begins, the teacher tells students specifically what strategies they are going to learn and 
why it is important for them to learn the strategies. The teacher then models how to use the 
strategies by thinking aloud with a text. Teacher-guided practice with feedback follows so 
students have opportunities to practice using the strategies. Next, the teacher provides 
independent practice opportunities for students to use the strategies, and discusses with time the 
strategies when they read on their own. 


Since some strategies are difficult to use with certain texts, PRJ carefully selects the text teachers 
use to teach each strategy. For example, main-idea summarizing is taught using informational 
texts, such as a content-area textbook or a nonfiction trade book. In addition, students study 
organizational text features that serve as frames for important information and logical links 
between ideas. Explicit instruction in text structures (sequence, compare/contrast, cause/effect, 
and problem/solution) teaches students to adjust comprehension strategies according to the text 
they are reading. Activities range from whole-group instruction to independent reading and 
response opportunities utilizing SOLO®. In each SOLO® session, students read leveled passages 
and practice previously taught strategies according to a specific learning path consistent with 
effective practices in computer-based instruction. 


Opportunities for Extended Discussion of Text Meaning and Interpretation. PRJ lessons 


engage students in discussions of the meaning and interpretation of lesson passages. These 
discussions occur mostly “during” and “after” reading, in whole groups, and in a non-threatening 
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and supportive environment under the guidance of the teacher. PRJ provides teachers with 
guiding questions to help them frame and extend discussions and to stimulate students to think 
reflectively about the passages. The questions give students opportunities to have sustained 
exchanges with the teacher and other students, present their interpretations and points of view, 
and listen to the reasoned arguments of others participating in the discussion. Students are 
encouraged to explain their positions and the reasoning behind them, propose counter arguments 
or positions, recognize good reasoning, and summarize the main ideas of a discussion as it draws 
to a close. Teachers gradually become more skilled at listening to and using learner responses to 
guide their next questions rather than vice versa. This models an effective thinking and 
questioning process that students themselves can use to lead quality discussions. 


Instruction in Foundational Reading Skills such as Decoding and Fluency. PRJ addresses 
word study in two ways. Students who read below 60 words per minute on a grade-level passage 
on PRJ’s Reading Connected Text measure are taught with an intensive word study program that 
begins with a review of single letter-sound correspondences. Lessons are provided in small 
groups and on alternate days in PRJ’s 10-day (two-week Expedition) lesson cycle. Explicit 
instruction focuses on automaticity and fluency with the alphabetic code. 


In addition, all students participating in the supplemental literacy intervention receive explicit 
and systematic instruction in decoding, with particular attention to the study of multisyllabic 
words and unfamiliar technical terminology, as well as systematic practice in reading fluently. 
Advanced Word Study lessons are incorporated into alternate lessons of the PRJ curriculum. The 
lessons provide instruction in affixes, sight words, decoding multisyllabic words, spelling, and 
word and phrase fluency. 


Instruction in Writing. PRJ includes two writing components. One component is writing in 
response to reading, which helps students check their understanding, reinforces returning to the 
text for more information, and sharpens critical thinking skills. Every second, fourth, and seventh 
lesson in a two-week reading Expedition integrates this component. The second component is a 
writing extension at the end of each Expedition designed to teach students to apply effective 
characteristics of good writing as they learn to write the text structures they are reading. These 
lessons help students develop writing proficiency essential for success in many content area 
classes. Students learn to apply qualities of good writing to their sentences, paragraphs, and short 
compositions. Instruction focuses on content, organization, voice, word choice, sentence fluency, 
and conventions. Paragraph writing is stressed, as well as multi-paragraph writing and research 
writing skills. Lessons employ explicit instruction, models of effective writing, and lesson- 
specific rubrics to enable self- and peer-evaluation. 


Strategies to Improve Student Motivation and Engagement in Reading. As students progress 
from elementary school to middle school, motivation to read often diminishes (Guthrie & Davis, 
2003). High interest text increases motivation to read as well as comprehension and achievement 
(Guthrie & Humenick, 2004). PRJ motivates students with engaging text and topics on different 
reading levels. Topics include real-world areas of interest for adolescents: Shark Attack, The 
Science of Catching Criminals, Predicting the Perfect Storm, How To Sweeten Your Ride, 
JEANology, Military Medics: Saving Lives in Battle, among others. The variety of topics and 
readability ranges allows students a considerable degree of choice. Audio recordings and online 
audio books are also available to provide models of prosody for striving readers. 
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DVD segments before and after each two-week reading Expedition present thought-provoking 
content in an action-packed format that provides background knowledge and creates the 
foundation for understanding of content. Each segment is hosted by a teen who poses probing 
questions, highlights essential content-area vocabulary, makes relevant connections to students’ 
lives and engages them in thinking about topics as far-ranging as the Internet, microscopic 
things, and money. The selections have been field-tested for high interest with middle school 
students and reach across the curriculum to foster literacy development in social studies and 
science. One of the most successful features is that the characters, content, and activities target 
students who represent diverse cultural and linguistic groups. 


Students also have access at least one day a week during school to SOLO® — an interactive web- 
based learning resource for students — which is also available to students whenever they can 
access it through the internet. This resource provides online reading resources matched to 
individual student reading levels as well as activities and assessments that provide immediate 
feedback to students which may increase their level of engagement and motivation to read. 


Reading Materials 


Description of Type of Texts Available.The PRJ student materials include several pieces. The 
Student Book includes student instructions, exercises, and worksheets. The Word Study Student 
Book contains lessons and daily passages for students who need additional instruction and 
reinforcement in phonic elements and sight words. The Student Assessment Book contains the 
tests and record sheets for the Reading Connected Text benchmark assessment. The benchmark 
assessment items for students can be located on the VPORT® Support Tab and the assessment 
can be taken online by logging into SOLO®. 


The additional pieces of the PRJ student materials that contain texts used by students include the 
Anthology, the PRJ Library, and SOLO®. The Anthology is a hardback book that is a collection 
of mostly expository reading passages used with the Expedition lessons. The Anthology contains 
approximately 5 to 7 passages for each of the 15 Expeditions. The PRJ Library contains 
magazines and books for students’ self-directed reading. Each level of the PRJ Library contains 
28 magazines and books focused on science and social studies topics. Finally, each SOLO® 
Expedition has up to four passages, mostly expository, which allow students additional practice 
of key comprehension and vocabulary skills. SOLO® also provides supplemental Book Cart 
articles that are accessible at any time during any SOLO® session. Students are encouraged to 
review Book Cart articles in their free time after the session work is completed. The Book Cart 
provides leveled articles related to the social studies and science topics found in PRJ 
Expeditions. 


Curriculum-Based or Teacher-Selected Materials. The Teacher’s Resource Kit contains all the 
lesson materials and other resources needed to deliver instruction. The Teacher’s Edition is the 
primary guide to teach the course. It contains daily lessons, examples of student materials, and 
facilitative dialogue to help teachers progress through each lesson. The Assessment Teacher’s 
Guide provides assessment and guidance about administering and scoring the Reading 
Benchmark assessment, Reading Connected Text assessments, Comprehension and Vocabulary 
Assessments, and SOLO® Self-Assessment. The Word Study Teacher’s Guide supports explicit, 
intensive instruction in fundamental reading skills. The Library Teacher’s Guide and Blackline 
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Masters manual provides an overview of the books and magazines in the PRJ Library. It is also 
the guide to student activities and blackline masters. The Transparencies Pack contains many 
graphic organizers for added visual support for vocabulary sections of the daily lessons. 
Expedition DVDs contain high-energy video openers that begin each Expedition. The video 
openers introduce students to the topic of the Expedition and pose interesting questions to engage 
and motivate students. These DVD segments contain a conclusion for each Expedition with 
suggested answers to the questions posed in the opener. 


Formative Assessment 


The Comprehension and Vocabulary Assessment at the end of each Expedition is the formative 
assessment included in PRJ. This assessment measures the comprehension and vocabulary skills 
that students have been taught throughout the Expedition. The test is referenced in the Teacher’s 
Edition and copies for students to complete are provided in the Student Book. The text used for 
the Comprehension and Vocabulary Assessments can be found in the Anthology. 


The Comprehension and Vocabulary Assessment helps to monitor each student’s acquisition of 
vocabulary and comprehension skills. It is written into instructional time to be administered 
during Lesson 9 of each Expedition. The assessment is administered to the whole group, lasts 
approximately 20 minutes, and simulates a high-stakes test format. 


Additional formative assessments are included in the SOLO® technology component. Practice 
sessions have real-time measures with automatic feedback for students. Teachers are able to 
review this student feedback in VPORT® under the SOLO® tab. 


Use of Technology 


Strategic Online Learning Opportunities or SOLO® is a technology tool through which 
instruction is delivered using a broad range of relevant and engaging topics. In each session, 
students are guided on the skill path of proficient readers by animated hosts who introduce the 
content, provide think-alouds that review the strategies taught in the classroom, and motivate 
students as they apply strategies to self-selected passages. 


SOLO® enables students to independently practice reading skills in an interactive, Web-based 
format. The program provides Lexile-leveled passages for students to read and explicit 
instruction and practice in word learning and comprehension strategies taught in the curriculum. 
Supports for independent reading include automated clues or prompts and vocabulary resources. 
The program also allows students to assess their fluency as they read a passage, review 
vocabulary words in a game format, and practice on skills in a standardized test format. SOLO® 
also connects students to additional technology resources for supplemental self-selected reading. 


Each Expedition contains two, 50 minute SOLO® sessions to be completed during the fifth and 
tenth days of the Expedition cycle. Time for SOLO® usage is built into the two-week Expedition 
cycle. 
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Implementation Study Design and Results 


Many of the specific assumptions of the research design in terms of anticipated effect size, 
sample size, outcomes of interest, and selected measures are based on the general assumption 
that the treatment is implemented with fidelity. SEDL developed methods to assess the fidelity of 
implementation in order to understand the degree to which PRJ was implemented as planned. 
SEDL worked closely with the curriculum developer during the first planning year of the study 
to develop and pilot test implementation fidelity measures and to measure features of high 
quality implementation of the two primary intervention components: professional development 
and classroom implementation. The research team collected data through classroom observations 
as well as interviews or surveys with teachers, administrators, district coordinators, and state and 
Voyager support staff as part of the implementation study. 


Research Questions 


The implementation study was designed to address the issue of fidelity overall by focusing on 
Research Question 1: 


Fidelity of Implementation (FOI. Describe the extent to which the intervention was delivered 
as the grant indicated it should be implemented. Was there significant variation in 
implementation fidelity among the schools and classrooms in the treatment group? 


The study focused on three areas to understand the degree to which the program was 
implemented with fidelity. Questions addressed in these areas included: 
Target Population: 


e Did the students, teachers, and schools in the PRJ program have the characteristics 
specified in the implementation model? 


Professional Development Model: 


¢ What was the nature of the professional development provided to teachers in PRJ 
schools? 


e¢ What type and amount of follow-up support/TA were provided to PRJ teachers? 


¢ How much variation of professional development, including follow-up support/TA, was 
there across districts, schools, and classrooms? 


Classroom Implementation Model: 


¢ To what extent to was the intervention delivered as the curriculum developers indicated it 
should be implemented? 


e¢ Was there significant variation in implementation fidelity among the schools and 
classrooms in the treatment group? 


¢ What were the major facilitators and barriers to implementation fidelity? 


SEDL | Research & Evaluation 21 


Louisiana Striving Readers: Final Evaluation Report 


Data Collection Plan 
Instrument Development 


Interview Protocols. Protocols were developed for use in interviews with intervention teachers, 
school administrators, Voyager coaches, and LDOE personnel (see Appendix B). Teacher 
interview protocols were designed to collect background information on the teachers and elicit 
comments about the various forms of training and support they received to implement the PRJ 
program, progress of implementation in their classrooms (including overall facilitators and 
barriers to FOI as they experienced them), and their perceptions of student response to PRJ. 
School administrators were asked to provide background information and comment on the types 
of support they offered to intervention teachers, their perceptions of support received from 
Voyager, district coordinators, and LDOE in implementing the program at their school, and 
challenges and successes experienced in their efforts to oversee PRJ classroom implementation 
as well as in their role as research study participants. 


Interviews with Voyager coaches focused on their background experience, their perceptions of 
the progress being made (and challenges experienced) by the intervention teachers they worked 
with directly, how PRJ implementation in the SR schools compared with the "typical" 
implementation in other districts they had observed, and their perceptions about the various 
forms of support being offered for the grant. Interview protocols developed for LDOE personnel 
asked about their background experience, their perceptions of how PRJ implementation was 
proceeding, and challenges/successes they were having in the process of supporting program and 
grant implementation. All of the interview protocols developed for and utilized in the 
implementation study were designed to address specific issues encompassed by the research 
questions used in the overall design of the study. 


Classroom Observation Instrument. The PRJ observation instrument (see Appendix B) was 
developed by SEDL specifically for this study and served as the primary measure of fidelity to 
the PRJ classroom implementation model defined by the developer. The development process 
began with the research team identifying the essential components of the PRJ program (i.e., 
those elements that are fundamental to implementing the curriculum according to the developer). 
Program components were originally identified through a thorough review of PRJ curriculum 
materials and Training Guides. Members of the research team also examined two of Voyager's 
observation tools: the PRJ Five Keys Reflection Tool, used by coaches during their classroom 
observations, and the PRJ Implementation Checklist, provided for teachers in the PRJ Training 
Manual. This review generated a list of 17 essential components of the program, further 
delineated into 40 sub-components. Eleven of these component categories focused on classroom 
delivery of the program and became the basis for developing the measure. 


The next step in development was to create a rubric aligning the degree of fidelity of 
implementation for the essential classroom components, measured on a 4-point scale (1 = poor, 2 
= fair, 3 = good, 4 = excellent). At the item level, the rubric included descriptions of 
implementation in each of the eleven primary categories, at each of the four levels. The 
descriptions provided observable characteristics of classroom implementation that would need to 
be present in order to assign scores at the various levels. The process of defining observable 
characteristics for each item was an iterative one with multiple stages, each of which was 
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reviewed by members of the research team. Once a draft of the instrument was completed, it 
was sent to Voyager to confirm that the correct classroom components had been included, and 
that item descriptions matched developer expectations for classroom implementation. 


The instrument was then piloted by members of the research team who would be conducting the 
classroom observations - first through independent viewings of a videotape from Voyager 
modeling high quality PRJ instruction, and later in actual PRJ classrooms in a local school 
district using the program. Following this pilot testing, the instrument was revised to improve its 
validity and reliability, and a training manual was developed for use in preparing observers to 
use the tool in LA Striving Readers study classrooms. 


Study data were collected by research team observers using the instrument in November, 2010. 
Observers went in pairs to each school and spent one day observing both 6th and 7th grade PRJ 
classroom instruction. The number of classrooms observed ranged from two to five for each 
teacher based on the number of sections they taught. The total number of observations completed 
in the fall of 2010 was 53. Consensus scores were calculated between each pair of observers for 
each observation. 


After further pilot testing and minor refinements, reliability on the instrument for the spring data 
collection had improved to 80% for two research team members, and these two individuals 
conducted observations in all of the LA SR Schools in March, 2011. A total of 20 classrooms 
were observed in the second round of data collection (one 6th and one 7th grade class for each 
teacher). Further testing was done on the PRJ Observation Instrument by conducting an 
exploratory factor analysis at the conclusion of first year data collection (discussed below). 


Data Collection 


Interviews were conducted in the fall and spring with teachers, Voyager coaches, and LDOE 
staff members to provide a variety of perspectives on the implementation process. Principals or 
other school administrators involved in the project were also interviewed in the spring. Teacher 
and administrator interviews were conducted in person at the schools, and Voyager and LDOE 
personnel were interviewed over the phone. A brief survey was administered online to get 
feedback from district coordinators and their role in the implementation of the program. 
Classroom observations were conducted in the fall and spring, and involved observing at least 
two classroom periods, followed by an interview with the teacher. The observations were 
scheduled well in advance in cooperation with the teachers and school liaisons to ensure that 
researchers were observing a typical day of class instruction and so that their visits to the school 
did not interfere with the coaches' site visit schedule. Members of the research team also 
attended the PD trainings conducted by Voyager to observe and collect data on what was 
presented, as well as participant responses. In addition, school and district records, classroom 
visit logs, and training attendance were provided for review by school liaisons, administrators, 
teachers, and district and state-level staff. Voyager agreed to make all VPORT records available 
to the research team. 


All data collection staff attended a training conducted by SEDL prior to the fall data collection 


period. A field procedures manual was developed with all the information needed for field staff 
to conduct the school visits, including an overview of all instruments and administration 
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procedures, as well as guidelines for appropriate behavior in the schools. Members of the field 
staff team who were not on the research team were introduced to the purpose and goals of the 
study and background information on the program. To ensure the ongoing quality of the data 
collected in the field, school visits were made by site coordinators to monitor field staff 
performance and provide feedback and additional training when necessary. School liaisons were 
trained at the beginning of the study to familiarize them with the research plan and their 
responsibilities, including procedures for managing data collection efforts at their schools. 


The primary types and sources of implementation data collected, along with the research 
questions they address, are summarized in Table 2.1. 


Table 2.1. Crosswalk of Research Questions on Implementation and Related Data Sources 
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Analysis Plan for Assessing Fidelity of Implementation 


Given that the goal of any implementation analysis is to describe the intervention, initial analytic 
efforts involved a strong mixed-method approach, using both quantitative and qualitative 
measures of implementation to describe the intervention as completely as possible. SEDL 
tabulated and described the type and amount of services delivered in the treatment condition 
using data from classroom observations, teacher interviews and other extant data sources. These 
tabulations used classroom or setting as the unit of analysis and were reported separately for 
participating schools. In addition, as discussed above, quantitative indices were developed to 
establish implementation benchmarks for the study regarding the extent to which the program 
was being implemented with fidelity in each treatment classroom and the length of time the 
program was implemented according to PRJ program standards. As a result, evidence from 
multiple methods and sources were used to triangulate on the concepts embedded in the program 
model to assess implementation fidelity. 


Overview of Analytic Plan 


The question of the degree of fidelity of implementation and the extent of variation by teacher 
and school was addressed several ways. Quantitative and qualitative mixed-methods approaches 
were applied to specific aspects of the research question and also used together to triangulate and 
interpret findings. Bulleted descriptions below briefly summarize the overall analytic approach. 


e Classroom observation scores were the core data source for the classroom FOI analysis; 
therefore, work began with the establishment of internal reliability for the PRJ 
observation instrument. 


¢ General descriptive data on FOI classroom observation codes for each teacher and class 
section observed were compiled. The data was also aggregated, in some cases across 
classes by teacher and by visit (fall and spring); 


¢ Comparisons of mean differences to determine the extent of classroom FOI as designed 
by end-of-study were conducted using total and subscales identified through factor 
analyses in paired sample t-tests; 


¢ Qualitative analysis of interview data was conducted to support and translate quantitative 
findings; target areas were teachers’ professional development experiences, site-specific 
information about facilitators of and barriers to fidelity of implementation, important 
contextual factors at sites, and variations in the quality of professional development 
delivered at sites; 


e Adequacy of classroom implementation fidelity was determined by a numeric threshold 
tied to observation score anchors (i.e., higher than threshold 2.5 on 1- to 4-point scale 
considered adequate FOI); 


e The professional development fidelity index was based on the number of PD training and 
support hours received by various recipients, as defined in the overall PD model 
established in the initial year of the study. Total FOI scores for each recipient were 
divided into high, medium, and low categories defined by pre-determined levels of 
adequacy of PD implementation. 
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Exploratory factor analysis: PRJ observation instrument reliability and validity. The 
analytic plan was designed to measure key components of fidelity of implementation to the 
intervention as designed. Classroom observation scores were the core data source for the 
classroom FOI analysis; therefore, work began with the establishment of internal reliability for 
the PRJ observation instrument. 


A preliminary phase of analysis was undertaken to establish internal consistency on observation 
scores according to key latent domains proposed to underlie the items. The observation 
instrument was designed to measure constructs generally conceptualized as adherence to 
instructional delivery materials and process that focuses on the quality of instructional delivery 
(O’Donnell, 2008). The individual items and the proposed domains had not undergone initial 
reliability or validity testing before data collection began. Under these conditions exploratory 
factor analysis (EFA) is recommended to determine underlying dimensionality and to establish 
baseline internal reliability of the data; principal axis factoring (PAF) and oblique rotation to 
allow correlation between the factors are the most commonly specified techniques for these 
purposes (Tabachnick & Fidell, 2001). 


Using data collected during the fall and spring semesters, the rated items were subjected to an 
EFA using PAF and oblique rotation. Each code of the instrument was considered the unit of 
analysis, providing a sample of 583 items, which corresponds to 11 items rated during each 
observation session across all classes observed at both time points for 9 teachers.’ The first run 
was not specified for any factor constraints, resulting in an unstable 4-factor solution that could 
not be interpreted but gave an indication that two factors may be emerging (i.e., based on 
inspection of the scree plot and the pattern of factor loadings). Constraining the extraction to a 2- 
factor solution, inspection of the pattern matrix revealed that the emerging factors were stable 
statistically and explained a substantial amount of the underlying meaning captured by the items 
(i.e., 55% of the cumulative variance). The pattern matrix was examined to determine reliability 
and inspected for item loadings that reached the .32 approximate cut-off threshold used to 
determine underlying factors and interpretability. Table 2.2 displays the item loadings for each 
factor. 


Table 2.2. Item Loadings from Exploratory Factor Analysis of PRJ Observation Instrument 


Observation Code Label Adherence (Loadings) Process (Loadings) 
1 Guide Ds) -.008 
2 Components 827 -.030 
8 Grouping 465 -.255 
3 Strategies .296 718 
4 Feedback -.032 655 
5 Pacing 563 47 
6 Involvement .099 147 
7 Monitoring -.048 .672 
9 Tasks -.130 a2 
10 Routines -.224 514 
11 Management -.111 835 


> One teacher’s observations were excluded because no data were available for fall and the factor scores were 
intended for comparisons between fall (baseline) and spring data. 
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Results indicated three items loading on Factor | that corresponded meaningfully to the label 
“Adherence,” the extent to which teachers used program materials in adherence with their 
intended design. The three items were “Guide,” the teacher’s close and consistent referencing of 
the curriculum guide to organize lesson content; “Components,” the teacher’s consistent use of 
lesson components described in the curriculum guide; and “Grouping,” which refers to whole, 
small, and individual or paired groupings used in instruction and guided practice for delivery of 
the program components. The item loading for “Guide” did not meet the cut-off point specified 
for this analysis (i.e., .32), but based on the pattern of item correlations and the interpretation of 
meaning for this item, it was retained. 


Results indicated seven items* loading on Factor 2, which was labeled “Process.” All of the items 
were coherent in content and conveyed teachers’ quality of delivery of the program as designed. 
The seven items described different components of the quality of program delivery: “Strategies,” 
the teacher’s use of reading strategies considered common in the field of reading instruction; 
“Feedback,” the positive or corrective feedback provided to students; “Involvement,” teachers 
ability to elicit student involvement by asking questions, encouraging participation, and making 
connections between the readings and students’ experiences; “Monitoring,” teachers’ awareness 
of student learning, understanding, and students’ progress on tasks or assignments; “Tasks,” the 
frequency of student engagement on tasks and their focus on class activities; “Routines,” 
students’ responsiveness to and transition between activities clearly established by daily program 
routines; and, “Management,” teachers’ actions related to running the class and managing 
student behavior. 


Factors emerging from the analysis were fairly distinct (r = -.07) and mapped onto theoretically 
derived constructs that differentiate the use of program materials from the processes involved in 
delivering the program and using instructional strategies, contributing initial verification of 
construct validity for the PRJ observation instrument. This distinction increased the specificity 
with which the question of level of implementation fidelity was addressed. The two-factor 
structure encompassed 10 of the 11 items in the full set (i.e., after dropping the item for pacing 
that did not load meaningfully) while partitioning them into two meaningful groups, thereby 
increasing the dimensionality within teacher (i.e., pre-post scores) and between teacher (i.e., 
scores may vary more on one domain than another). 


Year One Implementation Study Findings 
Treatment Schools 


Number and characteristics of treatment schools. Ten schools across four districts were 
recruited by LDOE to participate in the study. All schools met the inclusion criteria for the 
Striving Readers grant, and memoranda of understanding were arranged by LDOE and signed by 
principals at each participating school. Members of the research team also met with school and 
district administrators to explain the parameters and requirements of the proposed research and 


“ The results revealed that the item for “Pacing” double-loaded on the factors and therefore was dropped from 
further analyses. 
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to discuss procedures for minimizing disruptions to normal school routines during the study. 
Demographic characteristics for each participating school are shown in Table 2.3. 


Table 2.3. School Demographics 


SCHOOL 
1 2 3 4 5 6 7 8 9 10 

Total Middle 
School N=646 | N=606 | N=493] N=385 | N=588 | N=589 | N=545 | N=491 | N=342 | N=450 
Students 
Gender 

Male 52.5% | 52.6% | 51.9% | 47.0% | 50.3% | 53.0% | 51.6% | 51.7% | 51.5% | 49.6% 

Female 47.5% | 47.4% | 48.1% | 53.0% | 49.7% | 47.0% | 48.4% | 48.3% | 48.5% | 50.4% 
Race 

White 62.8% 7.4% 4.3% 1.0% | 14.6% | 35.8% | 62.0% | 20.2% | 27.2% | 46.4% 

Black 34.1% | 90.6% | 93.3% | 98.4% | 84.2% | 61.3% | 27.0% | 74.7% | 65.5% | 44.4% 

Hispanic 1.9% 0.8% 1.6% 0.0% 0.9% 2.4% 6.4% 2.4% 6.4% 8.4% 

Other 1.2% 1.2% 0.8% 0.5% 0.7% 0.5% 4.6% 2.6% 0.9% 0.7% 

Special Ed. 5.1% 6.8% 6.3% 9.1% 6.0% 5.4% 7.3% | 10.6% 2.9% 3.8% 

LEP 0.5% 0.0% 0.2% 0.0% 0.0% 0.8% 5.1% 0.8% 4.7% 4.2% 
Free/reduced 6 6 5 5 é Z 5 3 - 5 
Lunch Eligible 65.6% | 78.4% | 91.5% | 93.8% | 88.3% | 81.0% | 63.9% | 85.5% | 96.5% | 84.0% 


Interventionists Hired for Year 1 


Number. PRJ was implemented by literacy intervention teachers hired by the school districts 
based on selection criteria established by LDOE and provided below. LDOE worked closely 
with districts to hire one intervention teacher per school. Prior to the beginning of the school 

year, intervention teachers were in place across all of the 10 participating schools. 


Desired Characteristics of the interventionists for the SR project. LDOE determined the 
selection criteria and requirements for intervention teachers delivering the supplemental program 
activities during elective periods at each school. LDOE criteria and requirements, included: 1) 
valid Louisiana teaching certificate; meeting NCLB definition of Highly Qualified; 2) 
knowledge of reading intervention and scientifically-based research instructional strategies; 3) 
knowledge of specific strategies used before, during, and after reading to build comprehension; 
4) knowledge of strategies and activities to increase vocabulary and oral language development; 
5) ability to communicate effectively with students, school personnel, and parents, and an ability 
to work collaboratively with others; 6) ability to promote student motivation and engagement in 
learning; 7) knowledge of technology and how it is used to facilitate instruction and track student 
progress in reading; and 8) a minimum of three years of demonstrated effective classroom 
instruction, including teaching reading or core reading program and experience in improving 
student performance in diverse populations. In addition, an advanced degree in reading or 
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reading specialist certification was preferred. LDOE ensured the major responsibilities of the 
interventionist’s position were primarily related to the implementation of the PRJ program and 
the SR grant within the SR schools. 


Selection of Study Intervention Teachers. LDOE worked closely with districts to staff literacy 
intervention teachers who taught exclusively in the intervention classrooms rather than in 
addition to their duties in the regular ELA classrooms and to have one intervention teacher per 
school. Special care was also taken to ensure that treatment teachers did not interact with 
students assigned to the control condition. 


In the fall round of data collection, interviews were conducted with 9 out of the 10 intervention 
teachers due to the departure of one teacher prior to SEDL’s fall data collection visit. In the 
spring, information was gathered on the replacement for this intervention teacher. Based on 
demographic information gathered on 9 of the 10 teachers during fall interviews, all of the 
interviewed interventionists hired by the schools held valid Louisiana teacher certificates and 
met the NCLB definition of Highly Qualified.’ Table 2.4 provides demographic information on 
the intervention teachers’ qualifications and experience. 


Table 2.4. Demographics for PRJ Interventionists 


Years : 
Teacher sah Teaching oes Certification 
3 | ELA/Reading g 
1 6 2 BA Secondary Ed. 
English 
Reading 
Specialist/Ed 
: aa 2 Mn Leadership & 
Supervision 
3 15 5 MA Business/English 
4 28 2 MA Reading 
5 15 4 BA Language Arts 
6 Missing Missing Missing | Missing 
6 10 10 BA Reading 
7 9 4 MA Reading 
Language 
8 10 6 BS Arts/Elementary 
Ed. 
All subjects K- 
19 e ale 6/Reading K-12 
10 13 13 MA Reading 


In the spring, all 10 of the intervention teachers met these criteria as well. On average, the 
interventionists had 14.8 years of teaching experience (range of 6 to 32 years) across various 
grade levels and subjects, with an average of 6 years of teaching experience focused on reading 
(range of 2 to 13 years). Five out of the 10 intervention teachers had earned an advanced degree 
in a field related to education (i.e., Supervision, Reading, and Education). All 10 intervention 


> To be deemed highly qualified based on NCLB, teacher must have: 1) a bachelor’s degree; 2) full state 
certification or licensure, and 3) prove that they know each subject they teach. 
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teachers in the spring were certified in either Reading or English Language Arts demonstrating 
their knowledge related to the content area of the supplemental reading program. The 
intervention teachers, as a group, represent a highly qualified and experienced pool of teachers 
who met and exceeded the standards established by the LDOE for literacy intervention teachers 
hired to deliver the PRJ supplemental reading program. 


Turnover. One intervention teacher opted to leave her position mid-year (in November) due to 
challenges associated with implementing the program and for personal reasons (as reported by 
the school principal and district coordinator). Her departure occurred prior to the collection of 
demographic and implementation data. Shortly after her departure, the district coordinator 
collaborated with the school’s leadership to ensure that a qualified replacement intervention 
teacher was hired. The new intervention teacher’s demographic and implementation information 
was gathered during spring data collection visits and included in the description provided. 


Fidelity of Implementation of Professional Development for Interventionists (and 
Other School Staff) 


Professional Development Model. The professional development model delivered and/or 
facilitated by LDOE staff and Voyager implementation specialists included five formats for 
literacy intervention teachers and project support staff: Launch Training, Online Product 
Training, Ongoing Consultative Support, Cohort Meetings, and Principal Training. 


Assessing Implementation of Professional Development. Data assessing the implementation 
of the professional development model was collected through staff interviews, observations of 
professional development sessions, and record review (i.e., site visit logs, PD sign-in sheets, 
training agendas, and TA logs). These sources provided data to establish the implementation of 
the professional development model across the treatment schools and intervention teachers. 
SEDL researchers tracked Voyager’s PRJ program PD attendance through PD attendance 
records and site visit records completed by invention specialists. In addition to these measures, 
the evaluation team developed interviews for teachers, district and school staff, Voyager staff, 
and LDOE staff to collect data on the implementation of all aspects of the PD model, and 
VPORT was utilized to track intervention teachers’ participation in the online PD modules. 


PD Trainings 


Initial Cohort Meeting. During the first year of the grant a cohort meeting with key partners 
(i.e., LDOE staff, district and school staff, Voyager implementation specialists, and SEDL 
researchers) was held. The goals of the cohort meeting were to: 1) discuss programmatic 
activities related to the grant, 2) preview Voyager Passport Reading Journeys, and 3) review 
evaluation requirements. All of the participating districts were represented by their district 
coordinators on the first day of the training; however on the second day one of the district 
coordinators was unable to attend the meeting. Principals were also included in the intended 
audience for the information shared. On the first day of training, nine of the ten principals were 
in attendance and all were present on the second day of training. 


Teacher Launch Training. Initial group or “launch” training with intervention teachers, school 


principals, and district coordinators was planned over two days in early August 2010 with the 
intent of covering the key features of the PRJ program, including: assessment measures, student 
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grouping, classroom set-up, curriculum materials, implementation models, classroom 
management, and lesson delivery practice. Launch training was delivered by Voyager 
implementation specialists. SOLO® (Strategic Online Learning Opportunities) is part of the PRJ 
curriculum, which provides opportunities for students to practice skills learned during face-to- 
face instruction. VPORT® (Voyager’s online data management system) imports assessment data, 
monitors progress, and generates reports. 


In the fall, 9 of the 10 intervention teachers attended both days of the initial launch training. The 
teacher who was unable to attend the launch training had a scheduling conflict and could not 
attend; however, she completed the Online Product Training (OPT) prior to the first day of 
school. Based on fall interview data, the nine teachers who attended the launch training 
indicated they found it beneficial in understanding the PRJ program and some described it as 
‘more in-depth’ compared to other introductory trainings they had attended. Two of the 
intervention teachers felt that key information was glossed over (i.e., scoring assessments, timing 
of assessments vs. SOLO®). However, all of the teachers who attended the launch training felt 
adequately prepared to implement the program by the first day of school. 


In addition to the teachers, administrators from each of the ten schools and district 
representatives were also present at the initial launch training. Administrators from the schools 
provided support for the implementation of the Striving Readers grant across the school year and 
some aspects of implementation of the PRJ program. During spring interviews, the school 
administrators indicated the launch training had prepared their teachers for implementing the 
program and was helpful in establishing expectations. A few administrators acknowledged that 
their school had not been prepared to implement the program due to lack of resources (1.e., 
computers); however this limitation was not related to the training provided. 


Online Product Training (OPT). On VPORT®, under the Training tab, teachers were able to 
access the OPT modules. These are interactive, self-paced tutorials, which provide professional 
development targeted to the use of the PRJ curriculum. Teachers were required to log 8 hours of 
PRJ online training via the VPORT® system and complete 4 modules (i.e., Introduction, 
Curriculum Overview, Assessment Overview, and Implementation Overview). Upon completion 
of each module, the teacher received a certificate of completion. In addition, teachers were asked 
to participate in an additional 8 hours of online training on topics self-selected from a menu of 
available online support materials including video presentations, demonstrating before, during, 
and after reading strategies, advanced word study, reading, and classroom management. Based 
on the VPORT® system reports and teacher reports, by the spring, each of the 10 intervention 
teachers had completed her online training component. 


The online training had served as the launch training for the two intervention teachers who were 
unable to attend the two day in-person training help in August (1 had a schedule conflict and one 
was hired in November). One of the teachers felt the online training sufficiently prepared her to 
implement the training, while the other felt she would have benefited from additional support 
prior to implementing the program. It should be noted that the teacher who viewed the online 
training as insufficient was replacing an intervention teacher mid-year and was completing the 
online training and implementing the program simultaneously. Both of these teachers reported 
seeking out additional launch training opportunities; however, they were not able to attend a 
launch training due to scheduling conflicts. 
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The remaining intervention teachers (n = 8) had mixed reactions to the OPT modules. Two of 
them found the OPT helpful ‘as a refresher’ as they prepared for the second half of the school 
year. The remaining six teachers felt the OPT was repetitive of the fall training and not helpful. 
One teacher indicated that completing the OPT modules closer to the fall would have ‘helped 
reinforce the training’ versus waiting until December when she was already familiar with 
implementing the material. 


Cohort Meeting. In Spring 2011, a cohort meeting was conducted with the goal of bringing 
together all teachers and staff associated with the grant together to network, share successes and 
challenges, review data, and receive updates and information on program activities. During the 
first year of the project, the spring cohort meeting was attended by the 10 intervention teachers, 3 
district coordinators, 5 principals/school representatives, 2 LDOE staff, 3 SEDL evaluators, and 
6 Voyager PRJ representatives. 


The majority of the intervention teachers (9 out of 10) found the cohort meeting informative. 
They indicated the most productive aspect of the meeting was the data (i.e., Benchmark testing 
and VPORT data) shared during the presentations made by Voyager. The goal of data sharing 
was met, based upon teacher feedback. 


A second goal established by LDOE and Voyager for the meeting was the opportunity for 
attendees to network with one another and share successes and challenges. Three of the teachers 
indicated they had sufficient time to network; however, the remaining teachers described limited 
opportunities to interact with teachers, principals, and Voyager representatives. One teacher 
expressed the desire to have a Voyager coach’s perspective on her data in comparison to her 
principal’s; however, time was limited. Another teacher thought more discussion about 
implementation would have been beneficial. The one teacher who did not find the cohort 
meeting helpful indicated the presentation and data lacked clarity. 


Principal Meeting. A principal meeting was proposed to be held in year 2 of the grant. The 
purpose of the meeting in year 2 (fall 2010) was to explain the principal’s role in program 
implementation (e.g., walkthroughs, scheduling, technology needs). Due to budget limitations 
and travel restrictions this meeting was not held in the fall of year 2. 


Ongoing Support for Interventionists 


In addition to trainings, the PD model provided for several forms of classroom support for the 
interventionists, including various types of assistance from Voyager coaches, school 
administrators, district coordinators, and the LDOE. 


Coaching. Each teacher was assigned a coach from Voyager who was responsible for providing 
direct assistance with classroom implementation of the PRJ curriculum. Coaches scheduled 
monthly visits to each teacher's school to observe PRJ classroom instruction and to discuss any 
problems or issues the interventionist might be experiencing with program delivery. The original 
PD plan called for coaches to make 9 (2-4 hour) visits over the course of the school year. Each 
teacher received either 6 or 7 visits from their coach between October 2010 and May 2011, and 
visits generally lasted 1-3 hours. Conference calls with interventionists were conducted in 
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September, 2010, in lieu of classroom visits, due to the timing of the release of grant funding. 
Coaches also did not visit teachers' classrooms in March, 2011, because of the follow-up Cohort 
Training that month. During classroom visits, the coaches observed at least one PRJ class, 
scoring the teacher's level of implementation on Voyager's PRJ Implementation Checklist. 
Following the observation, the coach talked with the teacher about any issues she had noted 
during the observation and provided guidance about any problems the teacher might be having. 
In addition, the Voyager coaches were available to answer questions via email or phone between 
classroom visits. Voyager coaches assigned to the grant also conducted the Launch and Cohort 
PRJ trainings for teachers and administrators. 


School Administrators. Principals or other school administrators attended the PRJ launch 
training, the spring cohort training, and offered on-site assistance to their teachers to implement 
the curriculum. The majority of support offered by administrators was in the form of scheduling, 
providing equipment and supplies to the classroom interventionists, handling budgetary matters, 
and communicating with district and state representatives about various issues that arose during 
the first year of implementation. They were also helpful to the research team in scheduling site 
visits and arranging for student testing. Although the original professional development plan for 
the grant included regularly scheduled classroom observations by school administrators (1.e., up 
to 20 visits 90 minutes in length), Classroom visitor logs indicate that these observations did not 
occur in the majority of the SR schools. Classroom logs recorded multiple visits (12 throughout 
the year) by an administrator in one school, but most were brief "check-ins" rather than formal 
observations. Single classroom visits were recorded by administrators at three other schools - 
two at the request of the teacher and one for a general observation "required by the district." In 
interviews conducted by the research team, administrators reported that they had either done 
brief "walk through" observations of PRJ classes or more formal annual observations required by 
the district. These visits were not recorded in the Classroom logs, but most interventionists 
confirmed that they had occurred. In general, principals, assistant principals, and curriculum and 
instruction administrators in the schools offered more logistical support to the grant than 
instructional monitoring. 


District Coordinators. Support was also provided by district representatives who were 
designated by the grant to facilitate implementation in the schools. The type of assistance most 
frequently provided by the district was logistical in nature (e.g., providing needed equipment and 
supplies, handling budgetary issues, assisting school administrators with scheduling, and 
facilitating communication between the schools, the state, and the research staff). While district 
staff did attend the PRJ trainings in the fall and spring and were somewhat knowledgeable about 
the PRJ curriculum, their support of implementation was generally more technical in nature, 
rather than providing direct instructional assistance to the interventionists in the classroom. 
Classroom logs indicated that district representatives associated with the grant visited the 
classrooms of seven out of the ten interventionists and, when times were recorded, these visits 
lasted anywhere from ten minutes to two hours (average time recorded was 40 minutes). Nine of 
the total 14 visits documented were during the fall semester. District coordinators were also 
available to grant participants via phone and email communications. 


LDOE. Two representatives from LDOE were assigned to monitor and support the SR grant 
throughout the year. As with the administrators and district representatives, LDOE support 
consisted primarily of logistical, rather than substantive program assistance (e.g., the most 
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frequently requested form of support from LDOE was budgetary in nature). A total of ten 
classroom visits were recorded in Classroom Visitor Logs to eight of the ten interventionists. All 
but one of these visits were in the fall semester, and times recorded in the logs indicated that 
LDOE classroom observation visits lasted for an average time of about one hour and 15 minutes. 
State representatives indicated in interviews that classroom visits dropped off in the spring 
semester due to the demands of other duties in the LDOE office and budgetary limitations. Both 
LDOE monitors did, however, remain available to grant participants at all levels throughout the 
school year via phone and email. The state office also arranged and hosted the Launch and 
Cohort PRJ trainings. 


During the initial phase of planning, SEDL researchers collaborated with LDOE and Voyager to 
establish a professional development model, which outlined the format, purpose, and allocation 
of time for each type of professional development that teachers and school and district 
administrators would receive over the course of the grant, which was presented earlier. Table 2.5 
provides an abbreviated version of the intended PD model for all recipients for reference in this 
section. 


Table 2.5. Professional Development Model: Method, Frequency, Purpose, and Contact Hours® 


Method/Format Frequency Target/Audience 


School Administrators 
Initial Cohort Meeting Spring 2010 Districts Administrators 
LDOE 
Launch Training August 2010 Teachers 12 
School Administrators 
District Administrators 
LDOE 


Online Product Ongoing ¢ Teachers 8 to 16 
Training 


Ongoing Consultative 2x in September Teachers 18-36 
Support 1x in Oct. — Jan. 
1x in March — May 


2 to 4 hours per visit 


Cohort Meeting February 2011 ¢ Teachers 
School Administrators 
District Administrators 
LDOE 


° LDOE has enhanced Voyager’s PD model by providing additional supports highlighted in yellow, for each year of 
the study. 


Principal Meeting Fall 2010 ¢ _ Principals 
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Method/Format Frequency Target/Audience 


Total Hours (Teachers) 


It is important to note that the model was developed and agreed upon during the planning year of 
the grant by all partners: LDOE, Voyager and SEDL researchers. It was understood that the 
number of hours and the occurrence of professional development events would be tracked by 
SEDL researchers to accurately reflect the implementation of the grant and deviations from the 
planned PD model. In the analyses presented below, any deviations from the above model are 
presented and noted; however, overall levels of implementation are based upon the model as 
agreed upon during the planning year. In addition, hours have been combined across the planning 
year and the first year of implementation for the district and school administrators to determine 
implementation levels. 


Data were gathered across the first and second years of the grant via training sign-in sheets, 
classroom visits logs, teacher, administrator, LDOE, and Voyager coach interviews, as well as 
Voyager conference call notes. In the case of missing classroom visit logs, interview data was 
used to determine if the visits actually took place (i.e., teacher and Voyager coach reported visit 
occurred, but logs were missing). The number of hours were tracked for each teacher across the 
various types of professional development and an implementation index was then developed 
based upon the minimum number of hours identified in the professional development model (1.e., 
range of 8 to 16 hours for OPT - minimum of 8 was used to determine implementation index). 
The minimum number of required professional development hours for fidelity to implementation 
is 86 hours (26 training hours and 60 coaching/support hours) across the first year of 
implementation for teachers and 36 hours of training for school administrators. Applying the 
number of professional development hours actually received by teachers, levels of PD 
implementation fidelity are as follows: 0 = 51 or fewer hours (low); 1 = 52 to 68 hours 
(medium); and 2 = 69 or more hours (high). The proposed scales and cut-scores were based on 
previous Striving Readers grantees levels of implementation with the benchmarks of 80% of 
total hours or higher indicating “high” levels of implementation, 60% to 79% as “medium” 
levels of implementation, and less than 60% as “low” levels of implementation. Adequate 
amounts of PD were defined as having a high level of implementation of the PD model or having 
received 80% or more of the proposed PD hours. Table 2.6 focuses on the number of hours 
teachers participated in the different types of professional development provided during the first 
year of implementation: training and coaching support and the implementation level of the PD 
model and achieved adequacy. 
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Table 2.6. Teacher by Amount of Training and Coaching Support Received 


Total Number of Vevelot 
Teacher PD Hours Implementation 
Combined 

1 94 0 = Low 
D) 52 1 = Medium 
3 46.5 0= Low 
4 40 0 = Low 
5 36.5 0= Low 
6! 18.8 0 = Low 
Bs 17.5 0 = Low 
8 34.8 0 = Low 
9 39.8 0 = Low 
10 34.8 0= Low 
Ul 34.5 0= Low 


Only one of the ten teachers reached the level of medium implementation in relation to the 
professional development model when training and coaching support were combined. All of the 
teachers were below adequate, meaning they received less than 80% of the planned PD identified 
in the agreed upon model. 


In order to understand the challenge in implementing and receiving the planned professional 
development model, SEDL researchers opted to differentiate the types of professional 
development provided to teachers during the first year of implementation. Table 2.7 provides the 
number of training hours in comparison to the number of coaching support provided to PRJ 
intervention teachers, with levels of implementation (based upon the same scaling as detailed 
above 80% = high, 60% to 79% = medium and below 59% of hours = low) and adequacy. 


Table 2.7. Differential PD Hours by Training and Coaching Support 


Hours of Hours of : 
ae ae ; Coaching 
Teacher Training Training Coaching Support 
(Total Possible | Implementation | Support (Total Taolemantsdor 
26) Possible 60) P 
1 12 Low 20.50 0 = Low 
2 24 High 28.50 0 = Low 
3 24 High 23.00 0 = Low 
4 24 High 18.50 0 = Low 
5 24 High 13.00 0 = Low 
6” 12 High 15 0 = Low 


’ Teacher left school after third month of implementation. 

‘Classroom visit logs were not provided to SEDL researchers for one of the teachers, instead interview data was 
relied upon to provide this data — both the teacher and Voyager coach collaborated that the visits occurred and 
average length of the visit was used to represent missing data 

” Teacher left school after third month of implementation. 
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Hours of Hours of : 
ah ae : Coaching 
Teacher Training Training Coaching S <A 
ht (Total Possible | Implementation | Support (Total las eee an 
26) Possible 60) P 
ia 12 High 5.50 0 = Low 
8 24 High 11.25 0 = Low 
9 24 High 16.25 0 = Low 
10 24 High 11.25 0 = Low 
11 24 High 11.00 0 = Low 


When the types of professional development are separated, it becomes clear that ten of the eleven 
intervention teachers received adequate levels of training based on the professional development 
model outlined in the Louisiana Striving Readers grant. The one teacher who received a low 
adequacy level of training was unable to attend the launch training. 


None of the teachers received adequate levels of coaching support, as defined by the Louisiana 
Striving Readers’ grant professional development model. Within the model, coaching support 
was provided by multiple sources, including Voyager coaches, LDOE, and principals. A 
qualitative discussion was presented in the previous section that provides limitations and barriers 
to the implementation of the coaching support and the amount of time support was available to 
the intervention teachers. It should be noted that while the amount of time allocated for coaching 
supports was well below the adequate level, only the number of visits conducted by Voyager 
coaches was similar to that proposed within the model (average of 7 visits and 1 conference call 
in comparison to proposed 9 visits in the PD model). 


While teachers were the primary targets of professional development, within the implementation 
model proposed by Louisiana Striving Readers project key members of LDOE, district, and 
school staff also received professional development in order to support their teacher’s 
implementation of the PRJ program. An implementation index was also determined for the 
amount of PD principals and district representatives received. Table 2.8 provides the number of 
PD hours and the adequacy levels for district and school administrators. 


Table 2.8. School and District Administrators’ Professional Development Adequacy Level 


District : SNe, Implementation 
Administrator Serer: Level 
Hours 
1 25 2 = High 
2 25 2 = High 
3 21 1 = Medium 
4 25 2 = High 


'°Classroom visit logs were not provided to SEDL researchers for one of the teachers, instead interview data was 
relied upon to provide this data — both the teacher and Voyager coach collaborated that the visits occurred and 
average length of the visit was used to represent missing data 
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Professional 


School Development Implementation 
Administrator Hire Level 

1 25 1 = Medium 

2 25 1 = Medium 

3 25 1 = Medium 

4 25 1 = Medium 

5 15 0 = Low 

6 21 0 = Low 

vi 21 0 = Low 

8 25 1 = Medium 

9 19 0 = Low 

10 16 0 =Low 


The PD model was implemented at high levels for three of the four district administrators. 
School administrators were equally split between medium to low levels of implementation. This 
was primarily due to the cancellation of the principal meeting, which was meant to be held in the 
fall of the first year of implementation. The principal meeting was cancelled and not 
rescheduled; however, since it was part of the intended PD model the hours remained in the 
adequacy level reported here. If the 6 hours allocated to the principal meeting are removed from 
the calculation, then five of the school administrators would move to a high level of adequacy, 
three would move from low to medium and two would remain in the low category. 


Based upon the assigned implementation levels, the majority of the district coordinators received 
adequate levels of the intended professional development, while none of the school 
administrators received adequate levels of PD related to PRJ in order to implement the program 
as intended by the grant. 


Implementation of the Classroom Model 


Number and characteristics of students served. The total number of students served by the 
grant (i.e., were enrolled in PRJ classes at the beginning of the 2010-11 school year) was 617; of 
these, 288 were 6th graders and 329 were 7th graders. All of these students had scored in the 
Below Basic category in the ELA section of the 2010 7LEAP. Based on fall assessments using 
the GRADE measure (86% of students in the ITT treatment condition), the majority of students 
(79%) were two or more grade levels behind in reading which was the primary target population 
for the grant. Of the other students served, 15% were one grade level behind and 6% of students 
scored at or above reading level for their grade. Demographically, students in the treatment 
group did not differ significantly from those in the control group, and demographic data for all 
students in the ITT sample is presented in Table A.7 in Appendix A. The total sample of 
students served was relatively similar across the 10 schools and four districts served by the grant. 


Actual class sizes and student:teacher ratios. Each teacher's number of PRJ classes, class 
sizes, and average student:teacher ratios are shown in Table 2.9. The number of classes assigned 
to each teacher was determined by the number of eligible students randomized to treatment 
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classes in her school. Class size was limited to a maximum of 20 students, according to 
recommendations from the developer. Student:teacher ratios across all classes ranged from 19:1 
to 8:1, with an average of 13:1. 


Table 2.9. Class Sizes and Student:Teacher Ratios 


NUMBER AVERAGE 
TEACHER OF CLASS SIZES STUDENT: TEACHER 
CLASSES RATIOS 
i 6 15 | 13 | 13 | 13 | 12 | 12 13:1 
2 6 15/12} 10} 9 | 9 | 8 12:1 
3 4 13] 12] 9 | 6 10:1 
4 4 12} 8/7] 5 8:1 
5 4 17 | 16 | 13 | 10 14:1 
6 4 19 | 18 | 16 | 13 val 
7 5 18 | 15 | 14] 12} 5 13:1 
8 5 11/10} 9 | 8 | 7 9:1 
9 2 13] 9 11:1 
10 y) 19 | 19 19:1 


Actual intensity/dose of intervention students received. Several sources of data provide 
information about the dosage of intervention materials delivered. At the teacher level, available 
data include: class length, teacher attendance, and lesson pacing. The amount of class time 
available to the 10 interventionists in the study was fairly consistent. Nine out of the ten teachers 
taught class periods that lasted approximately 45 to 50 minutes. One teacher's classes were 75 
minutes long. Since PRJ is a supplemental program, elective periods were scheduled for 
treatment students rather than ELA blocks, which are normally 90 minutes. 


Teacher attendance across the school year is another possible indicator of dosage, especially 
since most teachers in the study reported that when they were not in class, PRJ materials were 
not used by substitute teachers. However, the data on teacher attendance collected is not reliable, 
since it was based on teacher recall of number of class days missed for all possible reasons (e.g., 
sickness, staff development days, PRJ training, non-curricular activities in the school building, 
etc.) over the entire year. The number of days that teachers reported being out of their classroom 
for the year ranged from four to twenty. 


Another measure of dosage for PRJ implementation is the number of lessons delivered in 
classrooms over the course of the academic year. Voyager defined the pacing schedule for the 
implementation of PRJ throughout the 2010-11 school year in study classrooms, and specified 
that the completion of 15 Expeditions (containing ten lessons each) constituted a "full dose" of 
the intervention. Data on each teacher's pacing came from two sources: what lesson they were 
teaching during the research team's spring classroom observations (in early May) and the number 
of Comprehension and Vocabulary Assessments (administered as a part of Lesson 9 in every 
Expedition) that were recorded for the year in VPORT®. There was agreement across these two 
sources of data on how many Expeditions each teacher completed over the course of the year, 
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and the final numbers ranged from 7 to 15, with an average number of 11.4 Expeditions 
completed. 


A final potential measure of dosage is student attendance. At the end of each month, school 
liaisons were asked to report on attendance for both treatment and control students. As students 
in the control condition were not in a specific alternative program, school liaisons reported on the 
number of days students in the control condition attended school for at least half of the day. For 
students in the treatment condition, school liaisons reported the number of days the students 
attended PRJ classes. Descriptive statistics for treatment students are provided in Table 2.10. 


Table 2.10. Treatment Students’ School Attendance, 2010-11 


School eens Mean Median 
oe 


Ptotar[——ar 0 | wos st tt | 


Based on Louisiana law, districts are required to provide at least 177 instructional days for 
students per school year. For a variety of reasons such as school absence, school assemblies, or 
attrition from the PRJ program, students in the treatment condition at the ten schools attended 
PRJ class 128 days on average. The median days of PRJ attendance was 151 (85% of the 
maximum) with a range from 119 to 165 median days. 


Student assessment procedures actually carried out. Delivery of PRJ curriculum materials is 
consistently informed by assessment procedures that are regularly scheduled throughout the 
school year. Primary assessments used include: 

e Reading benchmarks (RB) administered at the beginning, midpoint, and end of year 

e¢ Reading Connected Text (RCT) administered at the beginning, midpoint, and end of year 


¢ Comprehension and Vocabulary Assessment (CVA) administered after Lesson 9 of all 15 
Expeditions 


* SOLO® Student Self-Assessments -ongoing feedback provided in all SOLO® Lessons 


Student scores on each of these assessments are entered into the VPORT” online data 
management system and used to monitor student progress and assign students to appropriate 
Lexile-level texts. 
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Data provided online via Voyager's VPORT® system and from presentations at the Cohort 
Training are provided in Table 2.11 and show the number of teacher-administered assessments 
completed and recorded for each teacher by grade. All of the required RB and RCT assessments 
were administered and recorded by all ten teachers. The reliability of the CVA assessments is 
uncertain (i.e., inconsistency of scores may be due to a failure to record student results in 
VPORT®, rather than a failure to administer the assessments). 


Table 2.11. PRJ Assessments Completed 


ASSESSMENT AND TEACHER 
GRADE DEN ORAS eae | eae mor uleenel| estore NO 

Reading Benchmark (RB) [max = 3] 

6th 3 3 5 3 3 3 3 3 3 3 

(ia aes ee SSeS Ss es 
Reading Connected Text (RCT) [max = 3] 

6th 3 ) 3 g} 3 3 3 3 3 ) 

ies Be ae Se ee se Belle Bel Be s3 
Comprehension and Vocabulary Assessments (CVA) [max = 15] 

6" Ss Me 03 SO TE DD li), ES 

i Be eT pede TBs 8 oe TN et Dh ae nS 


Experience of control students. Students in the control group at 9 of the participating schools 
were enrolled in non-reading elective courses while treatment students attended PRJ classes. The 
elective courses offered across the schools varied (e.g., band, foreign languages, dance, art, 
chorus, physical education, and computers). The elective courses reflect the business as usual 
condition for the participating schools. Control group students at one of the participating schools 
were enrolled in a supplemental math program (TransMath). The supplemental math program 
was new to the school. 


Evidence of reading-related supports. In all 10 schools, additional reading-related supports 
were provided to students through their regular English Language Arts (ELA) curriculum via 
small group work. Homework and/or tutoring support before and after school across subject 
areas (i.e., math, science, social studies, and reading) was offered to all students in 6 out of 10 
participating schools. 


A smaller proportion of schools (4 out of 10) reported using Achieve 3000 as an additional 
reading support as part of their regular ELA classroom time. Information on the use of Achieve 
3000 was gathered from the principal at each school, and it was reported that all students were 
given the opportunity to interact with the program. The Accelerated Reader program was also 
implemented across all of the participating schools and was offered to all students. 
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Other Tier 2 Intervention Supports 


To address the growing need of students, all of the schools offered additional Tier 2 reading 
intervention supports to students not participating in the Louisiana Striving Readers project. 
Students receiving these services included those not selected to participate (i.e. scores too high), 
those who enrolled after randomization occurred, and those who did not have iLEAP scores. The 
Tier 2 interventions offered varied (i.e., Read 180, Earobics, Destination Read, Language!, and 
FastForward). It was most common for the schools to opt to implement multiple programs to 
accommodate various learning styles among their students. 


Fidelity of Implementation of Classroom Model 


A quantitative analysis of classroom implementation fidelity was conducted to identify levels of 
variation in implementation between teachers (schools) and across grades. 


Data Sources for Descriptive Analyses of Variation on PRJ Classroom Observation Scores. 
The 11-item observation scoring instrument used for each classroom observation session was 
rated on a scale from | to 4 points. The scores assigned by observers to each of the 11 items were 
summed for a “score total” and averaged for a “score average” for each classroom section 
observed. Teachers were observed multiple times (i.e., more than one class section per teacher) 
during the fall and spring semesters of data collection (n = 55 total observations). Table 2.12 
displays descriptive statistics for the individual items contained in the instrument as well as the 
total score and average total score. 


Table 2.12. Descriptive Statistics for Full Data Set PRJ Teacher Observations Scores 


N Range | Minimum | Maximum | Mean tice 

1 Guide 55 1 3 4 3.93 .262 
2 Components op 2 2 4 295 .803 
3 Strategies 55 2 a) 4 3.56 .631 
4 Feedback 55 2 zy 4 3.04 576 
5 Pacing 55 2 2 4 3:25 .673 
6 Involvement 55 2 2 4 3.29 .658 
7 Monitoring 3 2 2 4 3.35 .668 
8 Grouping ee) 1 3 4 3.9) .290 
9 Tasks a) 2 2 4 3.09 617 
10 Routines os) 2 Z 4 5.51 540 
11 Management 55 2 2 4 a2T od 

Score total 55 14 28 42 37.13 3.830 
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Score average 55 1 3 4 3.38 349 
Note: Overall reliability as measured by Cronbach’s Alpha was 0.77 


The full data set was split for fall and spring observation data and descriptives were run 
separately for each time point. Descriptive data for fall and spring score totals and score averages 
are displayed in Tables 2.13 and 2.14. 


Table 2.13. Descriptive Statistics for Fall 2010 Observation Codes" 


N Range |Minimum|Maximum} Mean Se 
Deviation 
Score total 31 13 2) 42 36.90 4.14 
Score average 31 1.18 2.64 3.82 3.36 38 
Table 2.14. Descriptive Statistics for Spring 2011 Observation Codes” 
oe : Std. 
N Range | Minimum} Maximum| Mean ie 
Deviation 
Score total 24 13 28 41 37.42 3.45 
Score average 24 1.18 2.55 3:13 3.40 RS | 


Descriptive analyses revealed non-significant variations between the data collected for each 
semester. Spring semester total score means were higher than those for fall, but the differences 
were not significant (¢ -1.17, p .26); differences between fall and spring semester score averages 
also were non-significant (t -1.16, p .26). 


Descriptive Analysis of Variation on Teacher-Level PRJ Observation Scores. The primary 
question of whether variation exists between teachers on the pattern of implementation fidelity 
scores is answered through several descriptive illustrations. Here, fidelity of implementation 
(FOD) is the extent to which teachers are observed using program materials and instructional 
strategies to deliver the program as designed. Observation score averages (i.e., the overall mean 
of scores averaged for each class section observed for each teacher) were used to operationalize 
teachers’ FOI of the classroom model. Figure 2.1 shows the between-teacher variation on 
average FOI scores for fall 2010 and spring 2011 observation data. The average FOI scores 
plotted for each teacher represent the mean averages for all observed classrooms’ average scores, 
merged for grades 6 and 7.The fall scores ranged from 2.66 to 3.73 and the spring scores ranged 
from 2.73 to 3.69. 


11 eae : : : 
Descriptive data for PRJ observation codes compiled from fall 2010 data contained consensus scores only and 
includes data from only 9 teachers due to teacher resignation in one of the schools at the time of fall data collection. 


o Descriptive data for PRJ observation codes compiled from spring 2011 data does not include consensus scores 
and includes all 10 teachers. 
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Figure 2.1. Fall and Spring Classroom FOI Averages by Teacher 
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Figure 2.2 displays the the between-teacher variation on average FOI scores for fall 2010 
observation data separated for 6" and 7" grades. 


Figure 2.2. Fall Classroom FOI Averages by Teacher and Grade.”° 
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The average FOI scores ranged from 2.69 to 3.82 for 6" grade and 2.64 to 3.82 for 7” grade ona 
1- to 4-point scale, with an average of 3.45 for 6" grade FOI scores and an average of 3.35 for 7” 
grade FOI scores for the 9 teachers for which fall data were collected. 


'’ The 7" grade average classroom FOI for teacher 5 is not reflected in the graphic because consensus scores were 
not available (i.e., data were collected by only one coder due to scheduling conflicts). 
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Figure 2.3 displays the between-teacher variation on average FOI scores for spring 2011 
observation data separated for 6" and 7" grades. 


Figure 2.3. Spring Classroom FOI Averages by Teacher and Grade. 
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The average FOI scores ranged from 2.55 to 3.73 for 6" grade and 2.91 to 3.73 for 7" grade on a 
1- to 4-point scale, with an average of 3.36 for 6" grade FOI scores and an average of 3.46 for 7” 
grade FOI scores for the 10 teachers for whom spring data were collected. 


Determining Extent of Fidelity of Implementation as Designed (Baseline to End of Study). 
In order to determine whether there were differences between baseline (fall) and end-of-study 
(spring) FOI scores, paired t-tests were used. The unit of analysis for this test was teacher, and 
given the small sample size, statistical power was limited. The comparison of means between the 
fall and spring FOI scores gives an indication of trends that could be detected in the extent of 
FOI achieved over the course of the one-year study. Table 2.15 shows the findings for the 
comparison of fall to spring means on the average total classroom observation scores. 


Table 2.15. Paired Samples T-Test Comparing Fall and Spring Average Total FOI Scores 


Paired Differences 
0 
Sid. | “Inewalofihe | + |at| Sf 
Mean SD Error : (2-tailed) 
Difference 
Mean 
Lower Upper 
Pair FOI Totals 
1 Averaged | -.90741 | 2.28917 | .76306 | -2.66702 | .85220 | -1.189)| 8 .268 
Fall & Spring 
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Results of the analysis indicated that there were no statistically significant differences between 
fall and spring scores 
(t -1.19, p .27). 


Factor scores from previous analyses of the classroom observation data (i.e., adherence and 
process) were analyzed using paired t-tests. Fall and spring adherence and process scores were 
computed by summing the average scores of the items loading on each factor and then averaging 
by the number of items representing the factor (1.e., 3 and 7) for each teacher. Figures 2.4 and 2.5 
show the average adherence and process scores for each teacher by semester. 


Figure 2.4. Fall Average Adherence and Process Scores by Teacher. 
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Figure 2.5. Spring Average Adherence and Process Scores by Teacher. 
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The average scores for items loading on Adherence and Process factors were then analyzed in a 
paired-samples t-test (two-tailed) to calculate the difference between means for the fall and 
spring FOI factor scores. Table 2.16 shows the findings for the comparison of factor means. 


Table 2.16. Paired Samples T-Test Comparing Fall and Spring Adherence and Process Scores 


Paired Differences 
0 
St eee ee aes 
Mean | SD | Error ; (2-tailed) 
Difference 
Mean 
Lower Upper 
Pair | Adherence 
1 | Fall & Spring 33 .46 15 -.025 .688 2.14 8 065 
Pair Process 
2 | Fall & Spring =27 .40 13 -.582 .039 -2.01 8 .079 


Results of the analysis indicated that there were no statistically significant differences between 
fall and spring scores on Adherence (f 2.14, p .07) and Process (¢ -2.01, p .08). Given these 
findings, the paired-samples t-test suggests that statistical power and the amount of time the 
teachers had to implement the program were both insufficient to conduct this analysis with any 
definitive findings. Taken together, however, the descriptive cross-sectional findings for fall and 
spring implementation revealed sufficient variation between time-points and grade-level 
implementation to suggest an underlying pattern that may have emerged more definitively over 
time and with a larger sample of teachers. 


Another set of paired-sample t-tests was conducted on the average scores for items loading on 
Adherence and Process factors but analyzed separately for 6" and 7" grade to calculate the 
difference between fall and spring FOI scores. Tables 2.17 and 2.18 show the results of the 
paired t-test for fall and spring factor scores, separated for grade. 


Table 2.17. Paired Samples T-Test Comparing Fall and Spring Adherence and Process Scores for 
6th Grade 


Paired Differences 


95% Confidence ‘ 
Be Interval of the t df Sig. 
Mean | SD Error : (2-tailed) 
Difference 
Mean 
Lower Upper 
Pair | Adherence 
1 | Fall & Spring 1.17 1.37 46 .114 2.219 2.55 8 034 
Pair Process 
2 | Fall & Spring -1.33 | 3.27 1.09 -3.846 1.179 -1.22 | 8 256 


Results of the analysis showed that there was a statistically significant difference between fall 
and spring scores when grade level was taken into account, but for Adherence only (¢ 2.56, p 
03), and only for 6" grade; no statistically significant difference was detected for the Process 
factor. 
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Table 2.18. Paired Samples T-Test Comparing Fall and Spring Adherence and Process Scores for 
7th Grade 


Paired Differences 


95% Confidence ‘ 
oe Interval of the t df aS 
Mean | SD | Error : (2-tailed) 
Difference 
Mean 
Lower Upper 
Pair| Adherence 
1 | Fall & Spring fa) 1.59 56 -.534 DAL 1.41 | 7 201 
Pair Process 
2 | Fall & Spring -2.56 | 3.32 | 1.17 -5.349 214 -2.18 | 7 .065 


As shown in Table 2.18, findings for the 7" grade were non-significant for both Adherence (t 
1.41, p .20) and Process (t -2.18, p .07) factors. 


Factors in Fidelity of Implementation 


Data related to the identification of primary factors affecting the overall level of classroom FOI 
during the first year came primarily from interviews with teachers, school administrators, 
Voyager coaches, and support staff at the LDOE. Additional contextual information was 
compiled from project files, researcher observations, and the results of a brief survey sent to 
district coordinators at the end of the first year. The interviewees provided a variety of 
perspectives on the primary facilitators and barriers to first year implementation of the PRJ 
program, depending upon their roles and positions in the effort. For instance, the teachers' 
(n=10) and coaches' (n=3) perspectives were largely focused on what happened in the classroom 
(i.e., implementation of the program), while comments from school administrators (n=10) and 
LDOE staff (n=2) focused on their particular roles outside of the classroom (i.e., implementation 
of the grant and/or the research study). The following discussion takes into account all of these 
perspectives and provides an overall summative account of the factors that most influenced 
classroom FOI during Year 1. 


Facilitators 


Support. There was general agreement across all informants that the most significant factor 
positively affecting program implementation was the strong support provided to the teachers and 
schools involved. Among the multiple forms of support provided to teachers, the most helpful 
was the guidance they received from Voyager coaches, who visited on a monthly basis to 
observe their classrooms and discuss any issues they might be having with their students or with 
technical aspects of implementing PRJ (e.g., conducting, recording, and using assessment data). 
The coaches, all of whom were working with other districts implementing PRJ, said that the 
amount of support provided to the SR interventionists was more than in their other districts. The 
grant itself allowed for this, since schools and districts are normally required to buy this level of 
support as an "add on" to the basic program package. Some of the teachers also specifically 
mentioned the support they received from networking with other interventionists either through 
informal communications (especially in one district where several teachers created their own 
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internal support network) or the opportunity to talk with others teachers at the Cohort Training in 
March 2011. 


School administrators received direct support primarily from the district-level coordinators 
designated by the SR grant to facilitate program implementation within each of the four districts. 
Forms of support most frequently requested and provided by the district representatives were: 1) 
procurement of materials and equipment needed for classroom implementation, 2) management 
of funding allocations for the schools, 3) assistance with scheduling treatment and control 
students, 4) coordination of release time for teachers and administrators to attend PRJ trainings, 
and 5) facilitation of communication between the schools, LDOE, and the research team. All of 
the district coordinators attended the Launch and Cohort Trainings and therefore had some 
content knowledge about PRJ, but their roles in supporting implementation were more 
facilitative than substantive in terms of direct assistance to the interventionists. As a group, they 
received support from staff members at the LDOE - most frequently in the form of assistance 
with budgetary issues. The general picture, then, is that the SR grant provided for multiple 
layers of assistance to support the implementation of PRJ, and that these various forms of 
support were a primary reason that a relatively high level of classroom FOI was achieved in all 
of the 10 schools involved. 


Curriculum. The second most important facilitator of implementation, according to the teachers 
and administrators interviewed, was the nature of the program itself. PRJ is a highly structured 
and scripted program that allows little room for personalized approaches to alternative 
implementation. Several teachers commented that it was easier than some other programs to 
implement with fidelity because of the extensiveness and coherence of the materials provided. 
The subject matter of the DVDs and readings were interesting to students overall, which 
increased their responsiveness in class and supported teachers' ability to implement the program 
correctly. Both teachers and administrators also mentioned that the highly structured nature of 
the program was particularly well suited for students who need supplemental assistance in 
reading. For instance, the (quick) pacing of material presentation and the variety of activities 
provided (including the SOLO® technology component) helped less advanced students stay 
engaged. 


Interventionists. A third reported facilitator of implementation was the quality of the teachers 
involved in the SR grant. The experience of the10 teachers implementing PRJ across all four 
districts ranged from 6 to 32 years, with an average of 14.8 years teaching experience overall. 
They were also highly trained - half had earned Master's degrees and all were certified in 
Reading, English, and/or Language Arts instruction. In addition, all of the teachers were 
specifically hired for, and funded by, the SR grant (and the majority did not teach any other 
classes in their schools). The importance of this factor was emphasized by one of the Voyager 
coaches who pointed out that many of the teachers she coached in implementing PRJ in districts 
outside of the grant were assigned to teach a section of PRJ in addition to their normal schedule, 
which often did not include reading or English instruction. These circumstances greatly 
enhanced the ability of the SR interventionists to focus and quickly become adept at 
implementing the program. 
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Barriers 


Pacing. Eight of the teachers and all three Voyager coaches interviewed reported that the 
biggest challenge they faced in implementing the program was the required pacing. All but one 
of the teachers were scheduled with approximately 45-50 minute classes, which was not enough 
time to cover all of the required components in a lesson (the one teacher who had longer class 
times - 75 minutes - did not experience this problem and also completed more Expeditions over 
the course of the year than the others). This barrier to implementation was more pronounced in 
the fall interview data, when the teachers were new to the program and trying to follow the 
instructions for pacing that they had received during the Launch Training at the beginning of the 
year. The problem was reduced for most when the coaches advised the teachers that they could 
spread the presentation of some lessons over two days when necessary. The expectation was that 
they would be able to "catch up" to the required pacing schedule over time by using parts of days 
assigned to SOLO® activities and re-teach days to cover material that hadn't been included 
during a previous lesson. Both teachers and students also became more adept at dealing with 
PRJ in-class pacing requirements as their experience with the program increased. 


Materials/Equipment. The second most frequently discussed barrier to implementation was the 
lack of sufficient materials - especially computers and other technological equipment or support 
(e.g., internet connectivity, Promethean Boards, or printers). The PRJ program requires that 
students complete one lesson per week on the computer (SOLO*), and the ideal circumstance is 
for teachers to be supplied with enough computers for all of their students' use in their classroom 
environment. Few of the SR interventionists were set up in this way, especially at the beginning 
of the year. Some reported having to use shared computer labs in their schools, which often 
meant that the SOLO® lesson was completed out of the correct sequence, since it was often not 
possible to reserve the lab when they needed it. Due to large classes and equipment shortages, 
other teachers had some computers in their classrooms, but not enough for all of the students. At 
the coaches' suggestion, they often devised ways to "rotate" computer use, with some students 
doing the SOLO® lesson while others did small group work, paired reading, or some other 
activity. While all of these solutions were ultimately effective, the general equipment problem 
was significant for the majority of the PRJ teachers. School administrators and district 
coordinators also reported being involved in trying to solve implementation issues around 
materials and insufficient equipment in several schools. 


Student Engagement. A third challenge to implementation discussed by nearly all teachers was 
maintaining student interest and engagement. As mentioned above, the content of the PRJ 
materials was reported to be effective at consistently engaging students. The challenges faced 
here were more contextual. For instance, because PRJ is a supplemental reading course, students 
in the treatment group were assigned to PRJ classes instead of their regularly scheduled 
electives, and there was some resistance to this, especially during the fall semester. Some 
teachers, administrators, and all of the coaches also noted that there were a few students included 
in the treatment group who were actually either too advanced or not advanced enough for PRJ. 
For instance, a few students in the sample did not have a strong enough understanding of basic 
reading concepts (such as phonics) to fully grasp PRJ content. Others demonstrated behavioral 
problems in the classroom significant enough to be an ongoing challenge for the teachers. Two 
teachers also mentioned having students that didn't really need PRJ, and as a result were 
consistently bored or distracted. The issue of student engagement is a common one, especially in 
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adolescent classroom settings. It is therefore not surprising that PRJ teachers and coaches 
identified it as a challenge to implementation, and interesting to note that overall they did not 
think it was due to the content of the PRJ materials. 


The major challenges discussed by school administrators had less to do with classroom program 
implementation and more to do with their role in implementing the grant and/or the research 
study. Participating in a research study inherently impacts the normal routine in a school to 
some degree, and where the administrators were most challenged was in the area of scheduling. 
The timing of student randomization for the research study (mid-summer) delayed class 
scheduling in some schools and, in some cases, created the need to re-schedule students who 
were incorrectly placed in treatment or control classes after the school year started. Several 
administrators also emphasized their concern with the fact that, due to the nature of the research 
study parameters, the control students in their schools were not able to benefit from the PRJ 
program. 


Conclusions 


Did students in the PRJ program receive the intervention as planned? The classroom FOI 
index was meant to evaluate the teacher’s overall score for delivering the intervention as 
planned, which stands as a proxy measure of whether the students received the intervention as 
planned. The index scores generated from the classroom observations were derived from spring 
data only. This decision was based on the need for a cumulative rather than a temporal measure 
of implementation. 


The data sources used to construct the classroom FOI index were translated into rubrics that were 
then compiled for an overall score for each teacher. The overall average observation scores were 
cut into ranges representing low, medium, and high categories of classroom FOI; low scores 
were assigned “0,” medium scores were given “1,” and high scores received “2.” Teachers’ FOI 
scores were considered low if they were on average below a score of “2.5” on the observation 
instrument — this threshold was determined from the actual PRJ observation instrument ratings, 
which were anchored by verbal descriptors that label score 2 “fair” and score 3 “good.” Based on 
these metrics, the 2.5 cutoff represents an adequate or mid-point level of program 
implementation. The range of scores corresponding to each index category was | to 2.5 low, 2.51 
to 3.44 medium, and 3.45 to 4 high. Forty percent of teachers were categorized medium and 60% 
were in the high category on average total FOI scores as presented in Table 2.19. 


Table 2.19. Summary of Classroom FOI Index Scores by Teacher 


Teacher 
Classroom Maximum Score | 1 | 2 | 3 | 4 | 5 6 7|8)} 9 | 10 
Average total FOI'* 2 ISR? ans | (ies? see se bet a Vee 


'* Average fidelity of implementation score across all class sections rated on 11-item observation instrument. 
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All teachers in this study achieved adequacy and a majority (60%) were implementing with high 
FOI index scores. The observation data used for the FOI index was merged on grade and on all 
classroom sections observed for each teacher. All 10 teachers were delivering the program at 
medium or high levels of implementation according to the fidelity index, which could be 
translated as an affirmative indication that the PRJ program was delivered with fidelity to 
students. 


How much variation in classroom fidelity of implementation was there between teachers 
(schools)? The question of how much variation in implementation fidelity was detected between 
teachers was answered through a series of descriptive data analyses comparing means for fall and 
spring observation data. Descriptive data showed that mean scores aggregated on grade and 
semester were not significantly different between teachers; the same finding held when data were 
disaggregated by semester. The average time between observations was five months, limiting the 
likelihood that differences would be detected between time points over such a short period of 
program implementation. Another possible interpretation is that teacher qualifications were so 
high overall that between-teacher variation on implementation was not likely to be significant. 


Overall, implementation fidelity was high and with limited variation between teachers, but some 
within-teacher variation existed in the factor scores to suggest a possible pattern. Findings from 
paired t-tests comparing fall to spring average implementation scores on “Adherence” and 
“Process” revealed a significant difference for scores on 6" grade Adherence, suggesting that 
teachers increased their fidelity to the use of PRJ 6" grade program materials as designed. 


Analysis of teacher interviews revealed a number of patterns that support and extend the 
interpretations of the quantitative patterns. Teachers mentioned the receipt of high levels of 
support to implement the program, corroborated by high attendance rates for professional 
development training. The program itself, especially in terms of its highly structured design, was 
reported by teachers as easy to implement. These specific program features, in combination with 
a highly qualified teacher sample, contributed to the overall high implementation scores. 


Did teachers in the PRJ program receive the intervention (professional development) as 
planned? Only one of the ten teachers received a medium level of PD (training and ongoing 
support) based upon the planned PD model for the grant, while the other nine were at a low level, 
and none reached the 80% designation for high or "adequate" PD. One explanation for these low 
overall PD FOI scores is evident when required PD hours are split between training and ongoing 
support/coaching. When the types of PD are separated, the data indicate that all but one of the 
teachers reached a high level of implementation in the training category, but all remained in the 
low category for coaching and ongoing support. The threshold for adequacy of PD support in the 
original PD model for the Striving Readers grant was set at a very high level which included a 
number of additions to a more standard definition of PD support for the PRJ program (e.g., 30 
additional hours for school administrator classroom observations and 12 additional hours for 
LDOE site visits). As noted previously, the time available for these additional forms of support 
was limited. 


Five of the ten school administrators reached a medium level of PD implementation, and five 


were in the low category. This was primarily due to the cancellation of a scheduled 6-hour 
training for administrators in the fall of 2010 that was included in the original PD model. When 
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these 6 hours are removed from the calculation, half of the administrators move to a high level of 
PD implementation, and three move from a low to medium level. Three of the four district 
coordinators reached a high level of PD implementation as defined by the grant. 


Overall, the PD FOI results indicate low to medium levels of implementation for the majority of 
recipients listed in the original PD model. It is unclear how much impact these ratings had on 
actual classroom implementation of the program, which was relatively high. 


How well was the intervention implemented, based on a PRJ Combined FOI index? The 
classroom FOI index scores presented above and scores for the professional development and 
support received by teachers were combined to create a more inclusive version of the index. The 
combined FOI index was drawn from multiple sources of data collected over the span of the 
study, including the amount of professional development delivered, ongoing support provided by 
the developer’s coaches, and observation scores of teachers’ program implementation. Scores 
used for the Professional Development portion of the combined FOI index were derived from the 
more extensive PD model discussed earlier. However, only those components of the model 
specified by the developer as training and support necessary for implementation of the PRJ 
program (i.e., not the SR grant) are included in the PRJ combined FOI index. Given these 
parameters, Professional Development components included in the combined FOI index are: a 
composite score for training (combined Launch, Cohort, and online trainings) and a total score 
for amount of Voyager coaching hours received by teachers. PD scores are based on the number 
of accumulated hours in each category for teachers discussed earlier, and assigned a 0-2 total 
value, according to the low, medium, and high adequacy ratings established for overall PD (0- 
.59=low adequacy or 0, .60-.79=medium adequacy or 1, and .80-1.0=high adequacy or 2). The 
classroom FOI components were calculated as above for the total, as well as the adherence and 
process factors, of the classroom implementation model. A total index score range of 0 to 10 
points is possible for each teacher and represents a cumulative measure for the combined PRJ 
FOI index (see Table 2.20). 


Table 2.20. Summary of PRJ Combined FOI Index Scores by Teacher 


Professional Development eonras aeacnee 

TNO E eS a ball Sorell ecole ee |e gE 
Training 2 DM Dee ae |e ly Qs all ele Weal ee Nye 
Support: On-site Coaching 2 Del hale- | C22] (22 | 307 NO 20 al] EL | 1 
Classroom 
Adherence 2 De a Pe ees Telly LS Me MeL eiie2 D 
Process 2 We ices ileal) a2]! Ge AL Ei 2-2 1 
Average Total Classroom FOI 2 Dts) dhe. |g 22 la ak UF se |e 
Total FOI score 10 Pe Nebel OR Ne Te sees ices alll veda ||| tenella 
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Since the scores for the PD and classroom fidelity indices shown in Table 2.20 were derived 
from different scales, the total FOI scores shown are cumulative and intended for descriptive and 
comparative purposes only. The combined PRJ FOI index scores indicate that 50% of teachers 
demonstrated 80% (8 out of 10) or higher levels of implementation, 20% demonstrated 70% 
levels of implementation, and 30% demonstrated 50% levels of implementation. These results 
indicate that relative to a more “typical” PRJ combined (professional development and 
classroom) implementation model, the program was implemented with medium to high levels of 
adequacy across the 10 schools in the study. 
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Impact Study Design and Results 
Impact Study Design 


The impact of the PRJ program on student motivation and reading outcomes is based on a multi- 
site randomized control trial study design. The RCT study design included student-level random 
assignment of eligible 6th and 7th grade students within participating schools. Students were 
randomly assigned to either a treatment group in which students received PRJ program services 
or to a control group in which students did not receive program services but may have received a 
range of other services available in their school (i.e., tutoring, study hall). The following sections 
reviews the design in more detail, including the research questions, sampling plan, random 
assignment plan, data collection, and impact analyses 


Research Questions 
The impact study addressed the following major research questions and hypotheses. 


Research Question: Does student-level assignment to the PRJ supplemental literacy program 
produce impacts on reading achievement for 6" and 7" grade students significantly behind their 
peers (i.e., scoring Below Basic on the state assessment) in reading relative to student assignment 
to a supplemental non-reading elective services control condition? 


Overall Impact on reading achievement 


PRJ has statistically significant effects on students’ reading achievement in 6-7" grade compared 
to the reading achievement outcomes of control students assigned to supplemental non-reading 
elective services. 


Overall Impact by specific reading subtest 


PRJ has statistically significant effects on specific student reading composite/subtest outcomes in 
6-7" grade compared to the reading achievement outcomes of control students assigned to 
supplemental non-reading elective services. 


Research Question: Does student-level assignment to the PRJ supplemental literacy program 
produce impacts on students’ motivation to read for 6" and 7" grade students significantly 
behind their peers (i.e., scoring Below Basic on the state assessment) in reading relative to 
student assignment to a supplemental non-reading elective services control condition? 


Overall Impact on motivation to read 


PRJ has statistically significant effects on student motivation to read outcomes in 6-7" grade 
compared to the motivation outcomes of control students assigned to supplemental non-reading 
elective services. 


Research Question: For which subgroups and under which conditions does the program have its 
greatest impact? 
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Student-level by poverty subgroup 


PRJ has statistically significant impacts on student reading achievement, specific student reading 
composite/subtests, and other student outcomes for 6"-7" grade students from low-income 
backgrounds compared to the outcomes of control students assigned to supplemental non-reading 
elective services. 


Student-level by ethnicity subgroup 


PRJ has statistically significant effects on student reading achievement, specific student reading 
composite/subtests, and other student outcomes for 6-7" grade students from ethnic minority 
backgrounds compared to the outcomes of control students assigned to supplemental non-reading 
elective services. 


Student- level by grade subgroup 


PRJ has statistically significant effects on student reading achievement, specific student reading 
composite/subtests, and other student outcomes in 6" and 7" grade compared to the outcomes of 
control students assigned to supplemental non-reading elective services. 


Student-level by gender subgroup 


PRJ has statistically significant effects on student reading achievement, specific student reading 
composite/subtests, and other student outcomes for 6-7" grade students for males compared to 
the outcomes of control students assigned to supplemental non-reading elective services 


Student-level by baseline proficiency in reading subgroup 


PRJ has statistically significant effects on student reading achievement, specific student reading 
composite/subtests, and other student outcomes for 6-7" grade students who enter the program 
reading significantly worse than their peers compared to the outcomes of control students 
assigned to supplemental non-reading elective services. 


Student Sampling Plan 
Random assignment procedures 


The SEDL data manager was responsible for procedures involved in the within-school random 
assignment of the eligible 6th and 7th grade students either to a treatment group in which 
students received PRJ program services or to a control group in which students did not receive 
program services but may have received a range of other services available in their school (ie., 
tutoring, study hall). Student eligibility criteria include recorded scores in the Below Basic 
category on previous spring administration of the LEAP ELA assessment. SEDL received and 
processed the 2010 iLEAP scores from LDOE staff in May. From June through August, SEDL 
staff worked with school and district personnel to obtain school rosters for the 6" and 7" grades. 
These rosters were then matched with the 2010 7LEAP results to create a list of students eligible 
for participation. 
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Based on rosters received, 1437 6" and 7" grade students at the ten participating schools scored 
Below Basic on the 2010 iLEAP and were randomized into the treatment and control conditions. 
Students were blocked by grade and school yielding an equal chance for all students to be 
selected into the treatment and control conditions. Analysis of mean differences provided 
evidence that the treatment and control groups were statistically similar in terms of demographic 
characteristics as well as reading and other outcomes (see Table A.7 in Appendix A). 


Students eligible for random assignment were those who scored Below Basic on the spring 2010 
iLEAP ELA subscale. During the first round of randomization, eligible students were sorted 
based on school and grade to establish a pool for each school prior to the first day of school. 
Eligible students were then assigned and sorted in ascending order by a random number to 
ensure they were not previously sorted by unknown variables (i.e., race, gender, free and reduced 
lunch status) for each grade in each school. From this list students were randomly assigned to 
first treatment and then control condition in order of the list (one to treatment, one to control) 
until the list of students had been exhausted. The SEDL data manager then assigned participant 
status based solely on ID number and sent the status directly to the site coordinator who in turn 
sent it to the school liaison and/or principal at each school. 


During the second round of randomization, SEDL worked with principals and school liaisons to 
identify an additional pool of eligible students who were not originally identified by early 
attendance rosters, but later enrolled. Student’s IDs were matched with the state data set to 
ensure they had not been randomized during the first round and to establish eligibility. 
Randomization procedures for the second round mirrored those used for the first round. Students 
were sorted by school and grade, then assigned a random number and sorted. Once sorted in 
ascending order by random number, students were randomly assigned to first treatment and then 
control conditions until the list of eligible students had been exhausted. As with the first round, 
randomization was performed by the data manager. Revised rosters were sent to the school 
liaisons and /or principals at each school. 


Approximately two weeks before the first day of school, rosters containing the names and state 
IDs for treatment students were sent to each school. Schools used this information to develop 
students’ schedules and plan for the number of PRJ sections required. In addition, the school 
liaison and/or principal reviewed the roster to confirm students were still registered at their 
school. After the second round of randomization, based upon updated rosters provided within the 
first week of classes, a revised roster was sent to schools. This version also contained 
information on students assigned to the control condition. The school liaison and/or principal 
reviewed this list and provided tracking information within the first two to three weeks of school. 


Numbers of students excluded from sample post RA and reasons why 


Though 1,437 students were randomized, 211 students were determined to be ineligible post- 
random assignment. Of these, 104 students had been randomized into the treatment condition 
and 107 students had been randomized into the control condition. As presented in Table 3.1, the 
majority of post-RA ineligible students (n=171 or 81%) never enrolled in the study school as 
anticipated by school personnel. Thirty-five (17%) students were also deemed ineligible for 
participation after they were promoted to 8" grade. Three students passed away over the summer, 
one student was homebound, and one student scored Below Basic and never attended a study 
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school but had a duplicate identification number to a student who scored Basic and enrolled at a 
study school. 


Table 3.1. Frequency of students discovered to be ineligible after random assignment 


Overall | Treatment | Control 

Promoted to 8th grade or higher 35 18 L7 

Students deceased 3 1 2 

Student not enrolled in study school 171 84 87 

Other 2 1 1 

Total 211 104 107 
Crossovers 


A common source of contamination is that a small but significant proportion of the sample in the 
control condition may receive similar services to the treatment condition students: Crossovers or 
students assigned to the control condition may enroll in a similar program or the same program 
in another school. One strategy used to address this was to maintain an active implementation 
component that monitored both conditions to detect ways that the intervention might be leaking 
and then address problems quickly. Perhaps more important than measuring implementation 
fidelity to safeguard against contamination, any randomization system needs to be closely 
supervised and leave no loopholes for deviations. Researchers provided school principals a roster 
of all students’ assignments as a way of tracking original randomized student groups and to 
provide a method for maintaining students in their original assigned groups if they transferred to 
other study schools. Students transferring to another study school were rostered to receive the 
intervention or to continue in control conditions, allowing them to remain in the original ITT 
sample. 


SEDL targeted resources to ensure that participants remained in their conditions over the length 
of the study. Monitoring the fidelity of randomization involved intensive and regular 
communication between site coordinators and school liaisons. Site coordinators worked with 
school liaisons to review participation, enrollment, and attendance records for students in both 
conditions. School liaisons also monitored vigilantly that students enrolled in specific electives 
in the control condition were attending those classes throughout the school year. 


When mistakes were discovered with student placement, site coordinators and school liaisons 
reviewed the records for each student and made a plan for correction based on collected 
information and on a case by case evaluation of the circumstances (e.g., how long the student has 
been in the wrong placement, how far into the school year before the mistake was discovered). 
During the course of the study, 3 students who were assigned to the control condition crossed 
over into PRJ treatment condition but were identified and moved out in less than one week. An 
additional 18 students from one district who were control students received other Tier-2 reading 
interventions (not PRJ) for, on average, less than 9 weeks. Taken together the study had a very 
small number of crossovers representing 3% of the ITT control group sample. 
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Description of ITT sample 


After students who were ineligible were removed from the study, the study sample included 
1226 eligible 6" and 7" grade students who scored Below Basic and attended the participating 
schools for at least one day. Six hundred nine students were randomized into the control group; 
284 6" graders and 325 7" graders. Six hundred seventeen students were randomized into the 
treatment group, 288 6" graders and 329 7" graders. 


As presented in Table 3.2, slightly more than half of the students in the ITT sample (57%) were 
male. Most students were Black or African American (71%), 24% of students were White or 
Caucasian, and 5% of students were Other (Hispanic, Asian, Native Alaskan, or two or more 
ethnicity/race categories). Approximately half of the students were in the 6" grade (49%). Fifteen 
percent of students were classified as Special Education students and 11% were classified as 
Section 504. Four percent of students had Limited English Proficiency (LEP) and 1% was 
classified as from migrant families. Nearly nine out of ten students were eligible for free or 
reduced price lunch (88%). 


Table 3.2. Description of ITT Sample 


Overall ITT Sample 
N = 1226 
N % 

Sex (Male) 696 56.8% 
Race 

Black/AA 865 70.6% 

White/Anglo 298 24.3% 

Other 63 5.1% 
Grade 

6 588 48.0% 

re 578 47.1% 

gh 60 4.9% 
Special education 186 15.2% 
Limited English 4A 3.6% 
proficiency 
Free/reduced lunch status 1075 87.7% 
Migrant 13 1.1% 
Section 504 Status 129 10.5% 
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Cases of Attrition 


The ITT sample included 1226 eligible 6" and 7" grade students though outcome data is not 
available for all students. Throughout the evaluation period during the 2010-2011 school year, 
incomplete data on the ITT sample can be attributed to missing pre- and post-test data as well as 
student attrition. 


School liaisons reported information to help track student attrition. This also helped ensure that 
any movement of a randomized student between participating schools would be caught and the 
student would continue in their original condition even across schools. The completed consort 
diagrams documenting attrition for both the state test ((LEAP), as well as the local evaluation 
baseline assessments (GRADE and MRQ) can be found in Tables A.2-A.4 in Appendix A. 


As presented in Table 3.3, during the course of the school year, students left and returned to the 
study schools for a variety of reasons which were classified into six categories: expulsion, parent 
opt out, removal from program by school administrators, transfer to a non-study school, 
promotion to 8" grade or higher, and other reasons. 


Most attrition events were triggered by students transferring to non-study schools after the first 
day of school. Although nearly 200 students had attrition events reported during the evaluation 
period, post-test data is available for many of these students who stayed within the Louisiana 
school system. 


Table 3.3. Frequency of attrition events as reported by school liaisons during school year 


Overall | Treatment | Control 


Expelled 2 3 2 
Parent opt out of PRJ participation 16 13 3 
Removed from PRJ by school 35 31 4 
administrators 

Transferred to another school 103 48 55 
Promoted to 8th grade or higher 16 10 6 
Other .) 2 1 
Total 178 107 71 


(14.5%) | (17.3%) | (11.7%) 


Missing Outcome (Pre- and Post-Test) Data 


The completeness of data is best described in two veins: one description for LEAP data and one 
description of GRADE and MRQ data. 
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iLEAP 


As eligibility criteria for inclusion in the evaluation depended on the spring 2010 iLEAP score, 
all 1226 students had completed pre-test data for the iLEAP. One thousand one hundred two 
students (90%) had post-test iILEAP scores. Seventy-seven students took alternative tests, based 
on promotion to 8" grade or other academic constraints, and 47 students had no test scores in the 
spring 2011 Louisiana database. 


GRADE & MRQ 


In fall 2010, pre-test data for the GRADE and MRQ were collected for all students in both the 
treatment and control conditions. Overall for both pre-tests, 1097 students completed the 
assessments, 532 treatment students and 565 control students. As presented in Tale 3.4, most 
students without pre-test data had moved to another non-study school. 


Table 3.4. Reason for missing pretest data for GRADE and MRQ (Fall 2010) 


Total | Treatment | Control 
Moved to non-SR school 41 17 24 
Unable to locate 30 19 11 
No test scores 15 8 
Parent opt out 18 16 
Other (SpEd, IEP, Ed Reqs) 25 24 
Total 129 84 45 


In spring 2011, post-test data for the GRADE and MRQ were collected for all students who were 
currently attending the participating schools. Nine hundred eighty three students completed the 
GRADE assessment. Nine hundred seventy seven students completed the MRQ assessment. 
Again, the most common reason for missing data was related to student mobility (see Tables 3.5 
and 3.6). 


Table 3.5. Reasons for missing posttest data for GRADE (Spring 2011) 


Total | Treatment | Control 
Moved to non-SR school 103 48 55 
Unable to locate, No test scores 62 23 39 
Promoted to 8" grade or higher 17 11 6 
Removed from PRJ by administrator 35 31 4 
Parent opted out of PRJ 16 13 3 
Expelled 5 3 2 
Other 3 2 1 
Total 241 131 110 
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Table 3.6. Reasons for missing posttest data for MRQ (Spring 2011) 


Total | Treatment | Control 
Moved to non-SR school 103 48 Do 
Unable to locate, No test scores 70 24 46 
Promoted to 8" grade or higher 17 11 6 
Removed from PRJ by administrator 25 31 4 
Parent opted out of PRJ 16 13 3 
Expelled 5 3 2 
Other 3 2 1 
Total 249 132 LZ 


Data Collection for the Evaluation of Student Impacts 


Researchers selected outcome measures for the study based upon their widespread use and 
reliability in impact studies of reading comprehension, including several recently funded by the 
Institute of Education Sciences (James-Burduny et al., 2009; Kemple et al., 2008) and the Office 
of Elementary and Secondary Education (Cantrell et al., 2008). The following section briefly 
describes instruments used in the study. 


Measures 


The data for the impact study relied on three student outcome measures: LEAP, GRADE, and 
MRQ. iLEAP and GRADE address the student reading assessment outcome domain and MRQ 
addresses the student efficacy, motivation, and engagement in reading outcome domain. 


iLEAP 


The Integrated Louisiana Educational Assessment Program ({LEAP) tests have been 
administered to students in grades 3, 5, 6 and 7 to test in English, math, science and social 
studies since 2006. The LEAP consists of norm referenced test (NRT) components 
supplemented with items developed to align with the Louisiana Grade-Level Expectations 
(GLEs). The additional GLE-based items combine with The Iowa Test of Basic Skills (ITBS) 
items that align with GLEs to form the criterion-referenced test (CRT) component of iLEAP. 
The difference between the two components (NRT and CRT) is the manner in which test results 
are interpreted. The two components yield two types of test scores: scores that represent 
students’ performance according to the Louisiana content standards (CRT scores) and scores that 
represent students’ performance compared to the national norms (NRT scores). Similar to state 
assessments, scores are measured in terms of the state’s achievement levels: Advanced, Mastery, 
Basic, Approaching Basic, and Unsatisfactory. 


The iLEAP assessment is carried out across the state by LDoE each year in March. After LDoE 
completed data collection and management for the LEAP, SEDL was sent a file with the scores 
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for all students in June 2010 for the pretest data and June 2011 for the post-test data. The data 
file included two test scores that were used in these analyses: the overall ELA scaled score and a 
reading scaled score that is a subtest of the ELA score. Results will be presented in both the 
original form of the scaled scores and converted to normal curve equivalents (NCEs). 


GRADE 


To capture a broader range of reading abilities, the evaluation included two additional 
assessments of reading: the GRADE and the MRQ. The Group Reading Assessment and 
Diagnostic Evaluation (GRADE; American Guidance Services, 2001) is a standardized norm- 
referenced, research-based reading assessment that can be administered to groups. The GRADE 
is organized into 11 levels of tasks or subtests that are designed to assess reading skill 
development from prekindergarten through early college. The GRADE contains a separate level 
for each year of school and each subtest is designed to measure skills that are developmentally 
appropriate for students at each level. Each level contains two equivalent forms to facilitate 
progress monitoring from fall to spring of each year and provides a normal curve equivalent 
(NCE) score in a standardized format. The GRADE measures four components of reading: 
reading readiness, vocabulary, reading comprehension, and oral language. For this study 
targeting grades 6 and 7, the focus is on vocabulary (nouns, verbs, adjectives, adverbs subtests), 
reading comprehension (sentence comprehension, passage comprehension, and metacognition 
subtests), oral language (listening comprehension subtest), and fluency (reading time, miscues, 
and comprehension subtests). Each individual GRADE level is designed to assess the reading 
skills of a wide range of performance at the recommended grade in school. Guidelines for out-of- 
level assessment are provided for striving students that may possess reading skills that are lower 
than what is typical of students at that grade level. 


The GRADE provides standard scores, percentile ranks, normal curve equivalents, stanines, and 
grade equivalent scores for individual subtests, composite scores, and total test score based on 
norms established for fall and spring at each level. The GRADE also provides a GSV score (total 
test score only) to track reading growth in students across years. In addition, the GRADE 
provides criterion-referenced scores for each of the subtests, categorizing student performance by 
number and percent correct as well as providing classroom profiles at the item level. A 
diagnostic analysis can be conducted in which a student’s item-by-item performance can be 
compared to a national sample or to peers in their classroom or in the study. Probability levels 
indicating the probability that a student will answer a particular item correctly can be developed 
for each student based on the national sample as well as for specific classroom or study samples. 


The GRADE is not a timed test and estimates of administration range from 45-90 minutes 
depending on the level. GRADE has strong evidence of reliability and validity for outcomes 
related to literacy and reading. Technical information compiled by AGS Publishing (2001) 
indicates the GRADE has a high degree of internal consistency for total, composite, and subtest 
scores for grades 6 and 7 (alphas between .95 and .98). Alternate form reliability was high (.88) 
and test-retest reliability coefficients were high (.90 and .94). Concurrent validity studies on 
grades 4-8 indicate moderate to strong correlations between GRADE and the Iowa Test of Basic 
Skills (.69-.83) and strong correlations with the Gates-MacGinitie Reading Tests (.87-.90) for 
grades 6 and 7. For these analyses, the NCE score were evaluated for the overall GRADE 
reading assessment, the vocabulary subtest, and the comprehension subtest. 
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The assessment was administered to the students twice during the 2010-2011 school year. The 
pre-test assessment of the GRADE was administered between the last week in August and the 
last week in September 2010 for all students. The post-test assessments were administered during 
the last two weeks of April and the first two weeks of May 2011. 


MRQ 


Researchers used a modified version of the Motivations for Reading Questionnaire (MRQ) 
(Cantrell, Almasi, Carter, & Rintamaa, 2008) to examine students’ intrinsic (e.g., curiosity, 
preference for challenging reading) and extrinsic (e.g., desire for recognition, reading for grades) 
motivation to read, reading involvement, and efficacy at using reading strategies (Baker & 
Wigfield, 1999). Confirmatory factor analyses have demonstrated that the student reading 
motivation subscales were reliable (Baker & Wigfield, 1999). Research has established the 
influence of students’ internal motivation, involvement or engagement in reading, and use of 
reading strategies on their improvement in reading comprehension (Guthrie et al., 2004; 
Taboada, Tonks, Wigfield, & Guthrie, 2009). 


The measure included 48 items that were answered on a one to four point Likert scale where 
students indicated how much they disagreed or agreed with each item. The students were not 
given a time limit to complete the measure. A confirmatory factor analysis was completed to 
match the four factor structure found by Wigfield and Guthrie (1995). The overall motivation 
score consists of a mean score of all 48 items. The subscales are mean scores of 17 items for 
intrinsic motivation, 11 items for extrinsic motivation, ten items for social motivation, and ten 
items for efficacy. The Cronbach alpha for the overall motivation score was 0.91 and the alphas 
for the subscales ranged from 0.71 to 0.78. 


The assessment was administered to the students twice during the 2010-2011 school year. The 
pre-test assessment of MRQ was administered between the last week in August and the last week 
in September 2010 for all students. The post-test assessments were administered during the last 
two weeks of April and the first two weeks of May 2011. 


Data Collection Methods 


This multi-year study required a comprehensive, well-defined, and at the same time flexible data 
collection plan. Data collection efforts involved two waves (fall and spring) over the data 
collection year. Each wave was organized around brief site visits to collect information from 
multiple sources using multiple methods. To implement this approach, SEDL created four major 
roles: 


¢ Data Manager. This experienced individual was the “director of operations” for the data 
collection plan and was responsible for implementing the plan and coordinating the 
efforts of key individuals and teams to complete all recruitment, random assignment, data 
collection, monitoring and quality control tasks for all field staff. 


e Site Coordinators. Two site coordinators were selected from among SEDL’s 
experienced research staff, with each coordinator assigned to two districts. Coordinators 
served as the primary study team contact for each school, assisting with final recruitment 
activities, facilitating random assignment, providing support for the school liaisons to 
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maintain participation in the study, and conducting quality control checks on field staff 
during data collection. It has been SEDL’s experience that staffing this position with 
PhDs who are trained to implement RCT designs with fidelity improves compliance with 
research methods and ultimately improves the quality of the data. 


¢ School Liaisons. A local staff person at each school was provided a stipend to support 
the site coordinators, helping them establish rapport with local staff and coordinating data 
collection efforts, obtaining informed consent, and scheduling interviews and other data 
collection activities during and after school visits. 


¢ Field Data Collectors. A group of six field data collectors under the supervision of the 
data manager was responsible for scheduled data collection activities for each wave, 
including conducting in-person student reading assessments and administering student 
surveys. 


Training and Quality Control. All data collection staff attended a two-day training conducted 
by SEDL prior to each data collection period. A field procedures manual was developed with all 
the information needed for field staff to conduct the school visits, including an overview of all 
instruments and administration procedures, as well as guidelines for appropriate behavior in the 
schools. Trainees were introduced to the purpose and goals of the study and background 
information on the program. Each field interviewer was given adequate practice and assessed to 
determine appropriate reliability levels on all instruments before being allowed to conduct school 
visits. Field interviewers also were trained to fill out data forms were checked in the field and 
then electronically scanned to ensure accuracy and efficiency. To ensure the ongoing quality of 
the data collected in the field, school visits were made by site coordinators to monitor field staff 
performance, re-check reliabilities in the field, and provide feedback and additional training, if 
necessary. School liaisons were trained at the beginning of the study to familiarize them with the 
study and their responsibilities, including procedures for recruiting students into the study and 
managing data collection efforts at their schools. 


Statistical Analysis of Impacts on Students 


Confirmatory and exploratory ITT analyses were conducted to address the two major research 
questions and their associated hypotheses: Overall program impacts and program impacts for 
particular subgroups. 


HLM Impact Analyses 


The principal goal of this evaluation is to investigate the effect of participating in the treatment 
(PRJ) versus business as usual for reading outcomes. All analyses are two-level (students within 
schools) fixed effects hierarchical linear models to account for variation in student and school 
level reading performance. Impact models include the fall pretest variable and a dummy variable 
for the treatment condition. Three main primary impact models are presented. 


> Overall Impact on reading achievement — PRJ has statistically significant effects on 
students’ reading achievement in 6-7th grade compared to the reading achievement 
outcomes of control students assigned to supplemental non-reading elective services. 


> Overall Impact by specific reading subtest — PRJ has statistically significant effects on 
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specific student reading composite/subtest outcomes in 6-7th grade compared to the 
reading achievement outcomes of control students assigned to supplemental non-reading 
elective services. 


> Overall Impact on motivation to read — PRJ has statistically significant effects on student 
motivation to read outcomes in 6-7th grade compared to the motivation outcomes of 
control students assigned to supplemental non-reading elective services. 


HLM Subgroup Analyses 


Several subgroup analyses were proposed to investigate for potential differences in the treatment 
effect between groups of students. 


> Student-level by poverty subgroup — PRJ has statistically significant impacts on student 
reading achievement, specific student reading composite/subtests, and other student 
outcomes for 6th-7th grade students from low-income backgrounds compared to the 
outcomes of control students assigned to supplemental non-reading elective services. 


> Student-level by ethnicity subgroup — PRJ has statistically significant effects on student 
reading achievement, specific student reading composite/subtests, and other student 
outcomes for 6-7th grade students from ethnic minority backgrounds compared to the 
outcomes of control students assigned to supplemental non-reading elective services. 


> Student- level by grade subgroup — PRJ has statistically significant effects on student 
reading achievement, specific student reading composite/subtests, and other student 
outcomes in 6th and 7" grade compared to the outcomes of control students assigned to 
supplemental non-reading elective services. 


>» Student-level by gender subgroup — PRJ has statistically significant effects on student 
reading achievement, specific student reading composite/subtests, and other student 
outcomes for 6-7th grade students for males compared to the outcomes of control 
students assigned to supplemental non-reading elective services 


>» Student-level by baseline proficiency in reading subgroup — PRJ has statistically 
significant effects on student reading achievement, specific student reading 
composite/subtests, and other student outcomes for 6-7th grade students who enter the 
program reading significantly worse than their peers compared to the outcomes of control 
students assigned to supplemental non-reading elective services. 


Year One Impact Study Findings 
Descriptive Analyses 


The following section provides the descriptive statistics, ITT impact findings, and subgroup 
findings for each outcome variable of interest for students participating in the evaluation. For 
each outcome variable, the descriptive statistics section provides a brief description of the 
variable, overall summary statistics, and results for treatment and control conditions. The 
descriptive statistics can be found in Table 3.7. 
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Table 3.7. Descriptive Statistics for Spring 2011 Outcome Variables 


Overall Treatment Condition Control Condition 


N Mean SD N Mean SD N Mean SD 


iLEAP State Test 


ELA Scaled Score 1102 | 253.16 | 42.30 | 548 | 253.93 | 42.61 | 554 | 252.41 | 42.02 
Reading Scaled Score 1102 | 252.09 | 56.83 | 548 | 251.41 | 58.57 | 554 | 252.76 | 55.10 
Below Basie 1102 | 77% | 0.42 | 548] 76%] 0.43 | 554]; 78% | 0.41 

GRADE 
Overall Standard Score 983 83.44 8.97 | 485 84.70 8.96 | 498 82.22 8.81 
Overall NCE 933 | 26.88 | 12.33 | 485 | 28.62 | 12.35 | 498 | 25.18 | 12.08 
Vocabulary Scale Score 933 | 86.92| 9.91 | 485 | 87.45] 9.94 | 498 | 86.40 | 9.87 
Vocabulary NCE 983 | 31.77 | 13.65 | 485 | 32.56 | 13.57 | 498 | 30.99 | 13.69 

eee Standard 983 | 82.95| 9.39 | 485 | 84.47] 9.36| 498 | 81.46] 9.18 
Comprehension NCE 983 | 26.26 | 12.79 | 485 | 28.33 | 12.91 | 498 | 24.25 | 12.35 


Motivation to Read 


Overall Motivation 977 2.76 | 0.44 | 484 2.78 | 0.41 | 493 2.75 | 0.46 
Intrinsic Motivation 977 2.82 | 0.41 | 484 2.84 | 0.39 | 493 2.81 | 0.43 
Extrinsic Motivation 977 2.86 | 0.56 | 484 2.87 | 0.54 | 493 2.86 | 0.58 
Social Motivation 977 2.52 | 0.56 | 484 2.54 | 0.54 | 493 2.51) 0.58 
Efficacy 977 2.80 | 0.54 | 484 2.82 | 0.52 | 493 2.77 | 0.56 


HLM Impact and Subgroup Analyses 


In this section, proposed models exploring treatment differences for subgroups will be discussed. 
HLM treatment effect estimates and effect sizes are presented in Table 3.8 below. The proposed 
subgroup exploratory analyses include models for gender, ethnicity, grade, poverty status, and 
pretest score. All other supporting detail, including HLM model specifications, tables with all 
predictors and estimations, statistical power analyses, effect size calculation, as well as 
discussions of treatment of covariates and missing data can be found in Appendix A. 
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Table 3.8. Overall Impact of the Intervention on Student Reading Achievement 


State Test Score - iLEAP 
Control Group Treatment Group 
Mean SD Mean Mean SD Mean Estimated Effect P-value 
(Scaled (Scaled (Converted (Scaled (Scaled (Converted Impact Size 
Score) Score) NCE) Score) Score) NCE) 

ELA Scaled Score 251.91 51.41 31.00 254.43 51.69 31.73 2.520 | 0.060 0.252 
Reading Scaled Score 252.23 73.90 27.83 251.94 74.30 27.48 -0.289 | -0.005 0.927 
Standardized Test - GRADE 

Control Group Treatment Group Scaled Score Model 
Mean SD Mean SD : 

(Scaled (Scaled ea ate (Scaled (Scaled ee ee ee aan P-value 

Score) Score) ( Score) Score) ( ) (Nee any gFeKe Was 
Overall Score 82.265 10.095 25.25 13.78 84.648 10.23 28.55 13.97 2.383 0.271 0.000 
Vocabulary 86.275 12.277 30.86 16.83 87.573 12.441 32.70 17.06 1.298 0.132 0.021 
Comprehension 81.568 11.333 24.38 15.17 84.364 11.485 28.20 15.37 2.796 | 0.304 0.000 

Motivation Test - MRQ 
Control Group Treatment Group Estimated | Effect 

i P-value 

Mean SD Mean SD Impact Size 
Overall Score 2.15 0.51 2.78 0.51 0.028 0.061 0.227 
Intrinsic 2.81 0.52 2.84 0.52 0.027 | 0.063 0.248 
Extrinsic 2.86 0.69 2.86 0.70 0.004 | 0.007 0.892 
Social 2.51 0.69 2.54 0.69 0.036 | 0.063 0.246 
Efficacy 2.77 0.65 2.82 0.65 0.043 0.078 0.143 
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iLEAP ELA Score 


Descriptive Statistics 


The iLEAP ELA score is computed from the ELA items that aligned with Louisiana's GLEs. For 
6" graders, the overall range of ELA scores is from 100 to 500 where any score below 280 is 
classified as Below Basic. For 7" graders, the overall range of ELA scores is also 100 to 500 
where any score below 286 is classified as Below Basic. Complete iLEAP data is available for 
1,102 students. All 1,102 students participating in this study scored Below Basic in their 2010 
ELA iLEAP tests. In the 2011 7LEAP, 77.2% of all study students scored Below Basic. In the 
treatment condition, 76% of the students scored Below Basic with an average ELA scaled score 
of 253.9. In the control condition, 78% of the students scored Below Basic with an average ELA 
scaled score of 252.4. 


iLEAP ELA Impact Model 


The Interclass Correlation ICC) was .012, indicating that approximately 1% of the variance in 
ELA scaled scores could be accounted for by school membership. 


In the two-level hierarchical linear model of iLEAP scaled score, there was not an overall 
significant treatment effect. As presented in Table 3.8 and Figure 3.1 below, after controlling for 
2010 ELA test scores and within-school differences, students in the treatment condition did not 
have significantly different scores than students in the control condition. Full model results are 
presented in Table A.8I in Appendix A. 
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Figure 3.1. Adjusted mean differences in iLEAP ELA NCE scores for treatment and control 
conditions 
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iLEAP ELA Subgroup Impact Models 


As presented in Tables A.8A-E (Appendix A), none of the five hierarchical linear models 
exploring subgroup differences in the treatment effect were significant. The treatment did not 
have differential effects on student iLEAP performance based on grade, gender, race, poverty 
status, or the 2010 pretest performance. 


iLEAP ELA Reading Subscale 


Descriptive Statistics 


The iLEAP ELA test actually consists of two subscales (Reading and Language) that are 
combined to give an overall ELA score. Given the hypothesized effects of the PRJ on students 
reading achievement, the Reading subscale, which is comprised by items on vocabulary and 
reading comprehension, was also analyzed for treatment effects. In 2010, 81% of students with 
complete LEAP data scored Below Basic on the Reading portion of the ELA iLEAP. In 2011, 
69% of students scored Below Basic with an average Reading scaled score of 252.1. 


In the treatment condition, 69% of the students scored Below Basic on the posttest with an 
average ELA scaled score of 251.4. In the control condition, 69% of the students scored Below 
Basic on the posttest with an average ELA scaled score of 252.8. 
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iLEAP ELA Reading Subscale Impact Model 


The Interclass Correlation ICC) was .027, indicating that approximately 3% of the variance in 
Reading scaled scores could be accounted for by school membership. In the two-level 
hierarchical linear model of Reading scaled score, there was no significant treatment effect (see 
Table 3.8 and Figure 3.2 below). Students in the treatment and control conditions did not have 
significantly different Reading scores. Full model results are presented in Table A.9I in 
Appendix A. 


Figure 3.2. Adjusted mean differences in iLEAP Reading NCE scores for treatment and control 
conditions 
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iLEAP ELA Reading Subscale Subgroup Impact Models 


As presented in Tables A.9A-E (Appendix A) and Figure 3.3 below, only one of the proposed 
exploratory analyses of subgroup differences resulted in a significant treatment difference, 
comparisons of the treatment effect for 6" and 7" graders. Seventh graders in the treatment 
condition scored significantly higher than 7" grade students in the control condition. Sixth 
graders in the treatment condition scored significantly lower than their counterparts in the control 
condition. 
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Figure 3.3. Adjusted mean differences in iLEAP ELA NCE scores for treatment and control 
conditions by grade 
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GRADE Overall Reading Score 


Descriptive Statistics 


Of the 1226 participating students, 983 had valid posttest scores for the GRADE assessment. The 
GRADE assessment produces the standardized score and the normal curve equivalent (NCE) for 
an overall reading score and for vocabulary and comprehension subscales. This section includes 
the results for the overall reading scores; vocabulary and comprehension subscales are covered in 
following sections. 


Students in the treatment condition had an average NCE of 28.6 and standard score of 84.7 while 
students in the treatment group had an average NCE of 25.2 and standard score of 82.2. 


GRADE Overall Reading Score Impact Model 


The ICC value for the overall GRADE was 0.02. In the HLM model, the treatment effect on the 
overall GRADE was significant (see Table 3.8 and Figure 3.4 below). Students in the treatment 
condition scored significantly higher on the GRADE than students in the control group. The 
adjusted mean NCE for treatment students was 28.6 while only 25.2 for control students. The 
adjusted mean standard scores for treatment students were 84.6 and 82.3 for control students. 
The effect size was 0.27. Full model results are presented in Table A.10I (standard score) and 
A.111 (NCE score) in Appendix A. 


SEDL | Research & Evaluation 72 


Louisiana Striving Readers: Final Evaluation Report 


Figure 3.4. Adjusted mean differences in GRADE NCE for treatment and control conditions 


35 


w 
oO 


28.55 


NCE 
(Adjusted Means) 


25.25 


N 
uw 


20 
Control Treatment 


Treatment Condition 


GRADE Overall Reading Score Subgroup Impact Models 


Of the five proposed subgroup models for the overall GRADE NCE, there were significant 
interactions between the treatment condition and grade and gender (see Tables A.1O0A-E 
(standard score) and Tables A.11A-E (NCE score) in Appendix A and Figures 3.5 and 3.6 
below). For gender, the treatment had little effect on females. However, males in the treatment 
condition scored significantly better than males in the control group. 
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Figure 3.5. Adjusted mean differences in GRADE NCE scores for treatment and control 
conditions by gender 
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Similar results existed by grade. For grade, the treatment also had little effect on 6" graders 
reading. For 7" graders though, students in the treatment condition scored significantly better 
than students in the control condition. 


Figure 3.6. Non-adjusted mean differences in GRADE NCE scores for treatment and control 
conditions by grade 


@Grade6 


® Grade 7 


Overall GRADE NCE 


0 + 


Control Treatment 


GRADE Vocabulary Subscale Score 


Descriptive Statistics 


Students in the treatment condition had an average NCE of 32.6 and standard score of 87.5 and 
students in the control condition had an average NCE of 31.0 and standard score of 86.4. 
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GRADE Vocabulary Subscale Score Impact Model 


The treatment effect for student impacts on GRADE vocabulary subscale scores was statistically 
significant (see Table 3.8 and Figure 3.7 below). The adjusted NCE for the treatment condition 
was 32.7 and the adjusted NCE for the control condition was 30.9. For the standard score, the 
adjusted mean for the treatment condition was 87.6 and 86.3 for the control condition. The 
treatment had a positive and significant effect on students' vocabulary performance. The effect 
size was 0.13. Full model results are presented in Table A.12I (scale score) and A.13I (NCE 
score) in Appendix A. 


Figure 3.7. Adjusted mean differences in GRADE Vocabulary NCE scores for treatment and 
control conditions 
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GRADE Vocabulary Subscale Score Subgroup Impact Models 


None of the subgroup models exploring differential treatment effects produced any significant 
results (see Tables A12A-E (standard score) and Tables A13A-E (NCE score) in Appendix A). 


GRADE Reading Comprehension Subscale Score 


Descriptive Statistics 


The comprehension subscale consists of measures of Listening, Passage, and Reading 
Comprehension. Students in the treatment condition had an average NCE of 28.3 and students in 
the control group had an average NCE of 24.2. For the standardized score, students in the 
treatment students had an average score of 84.5 and 81.5 for students in the control condition. 
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GRADE Reading Comprehension Subscale Impact Model 


The HLM model also produced evidence of a significant treatment effect (see Table 3.8). 
Students in the treatment condition scored significantly better than students in the control group. 
The effect size was .31 and the adjusted mean difference in NCE across treatment and control 
conditions was 3.8 (see Figures 3.8 below). The adjusted mean standard score for the treatment 
condition was 84.4 and 81.6 for the control condition. Full model results are presented in Table 
A.14I (standard score) and A.15I (NCE score) in Appendix A. 


Figure 3.8. Adjusted mean differences in GRADE Comprehension NCE scores for treatment and 
control conditions 
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GRADE Reading Comprehension Subscale Subgroup Impact Models 


As presented in Tables A.14A-E (standard score) and Tables A.15A-E (NCE score) in Appendix 
A and Figures 3.9 and 3.10 below, two of the subgroup HLM models revealed differential 
treatment effects. For gender, the effect of the treatment was positive for all students but 
strongest for males. For males, students in the treatment condition scored significantly better 
than students in the control condition. The treatment had little effect on females’ scores. 
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Figure 3.9. Adjusted mean differences in GRADE Comprehension NCE scores for treatment and 
control conditions by gender 
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Similar results existed by grade for the Comprehension subscale. The treatment had little effect 
on female students’ scores. However, the treatment had significant effects for male students. For 
males, students in the treatment condition scored significantly higher than students in the control 
condition. 


Figure 3.10. Adjusted mean differences in GRADE Comprehension NCE scores for treatment and 
control conditions by grade 
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There was also a significant interaction effect between pretest score and treatment condition for 
GRADE Comprehension, although only for the standardized score model and not the NCE 
model. The significant interaction term indicates that students with higher pretest scores had 
higher posttest scores; however, this relationship was stronger for treatment students (see Figure 
3.11). Given the lack of consistency of this finding between two different measures of the same 
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subscale, this finding may be a statistical artifact of some kind, so it will not be included it in the 
summary of findings. 


Figure 3.11. Adjusted mean differences in GRADE Comprehension scores for treatment and 
control conditions with pretest interaction 
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Motivation to Read Questionnaire: Overall Score 


Descriptive Statistics 


Overall, students in both the treatment and control conditions reported high levels of motivation 
to read. Nine hundred seventy seven students completed the MRQ at the posttest. On a scale of 
one to four, students in the treatment condition had an average score of 2.78. Students in the 
control condition had an average score of 2.75. 


MRQ Impact Model 


There was no significant treatment effect on students' overall motivation to read as presented in 
Table 3.8 and Figure 3.12 below. Full model results are presented in Table A.16I in Appendix A. 
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Figure 3.12. Adjusted mean differences in MRQ scores for treatment and control conditions 
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MRQ Subgroup Impact Models 

There were no significant interactions of the treatment effect by student demographics on 
students' overall motivation to read (see Tables A.16A-E in Appendix A). 

Motivation to Read Questionnaire: Intrinsic Motivation Subscale 


Descriptive Statistics 

On a scale of one to four, students in the treatment condition had an average score of 2.84. 
Students in the control condition had an average score of 2.82. 

MRQ Intrinsic Subscale Impact Model 


As presented in Table 3.8 and Figure 3.13 below, there was no significant treatment effect on 
students' intrinsic motivation. Full model results are presented in Table A.17I in Appendix A. 
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Figure 3.13. Adjusted mean differences in MRQ Intrinsic Motivation scores for treatment and 
control conditions 
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MRQ Intrinsic Subscale Subgroup Impact Models 


There were no significant interactions of the treatment effect by student demographics on 
students' intrinsic motivation (see Tables A.17A-E in Appendix A). 


Motivation to Read Questionnaire: Extrinsic Motivation Subscale 


Descriptive Statistics 


On a scale of one to four, students in the treatment condition had an average score of 2.87. 
Students in the control condition had an average score of 2.86. 


MRQ Extrinsic Subscale Impact Model 


There was no significant treatment effect on students' extrinsic motivation as presented in Table 
3.8 and Figure 3.14 below. Full model results are presented in Table A.18I in Appendix A. 
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Figure 3.14. Adjusted mean differences in MRQ Extrinsic Motivation scores for treatment and 
control conditions 
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MRQ Extrinsic Subscale Subgroup Impact Models 


There were no significant interactions of the treatment effect by student demographics on 
students' extrinsic motivation as presented in Tables A.18A-E in Appendix A. 


Motivation to Read Questionnaire: Social Motivation Subscale 


Descriptive Statistics 


On a scale of one to four, students in the treatment condition had an average score of 2.54. 
Students in the control condition had an average score of 2.51. 


MRQ Social Motivation Subscale Impact Model 


As presented in Table 3.8 and Figure 3.15 below, there was no significant treatment effect on 
students' social motivation. Full model results are presented in Table A.19I in Appendix A. 
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Figure 3.15. Adjusted mean differences in MRQ Social Motivation scores for treatment and 
control conditions 
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MRQ Social Motivation Subscale Subgroup Impact Models 


There were no significant interactions of the treatment effect by student demographics on 
students' social motivation (see Tables A.19A-E in Appendix A). 


Motivation to Read Questionnaire: Efficacy Subscale 


Descriptive Statistics 


On a scale of one to four, students in the treatment condition had an average score of 2.82. 
Students in the control condition had an average score of 2.77. 


MRQ Efficacy Subscale Impact Model 


There was no significant treatment effect on students' efficacy as presented in Table 3.8 and 
Figure 3.16 below. Full model results are presented in Table A.20I in Appendix A. 
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Figure 3.16. Adjusted mean differences in MRQ Efficacy scores for treatment and control 
conditions 


Scale Score 
Ww 


(Adjusted Means) 


N 


Control Treatment 


Treatment Condition 


MRQ Efficacy Subscale Subgroup Impact Models 


There were no significant interactions of the treatment effect by student demographics on 
students' efficacy (see Tables A.20A-E in Appendix A). 


Conclusions 


What were the ITT sample characteristics? The ITT analytic sample included 1226 6" and 7" 
grade students from 10 middle schools across the state of Louisiana that scored Below Basic on 
the state assessment in reading. The students assigned to the treatment and control condition in 
the ITT sample were largely male (57%), economically disadvantaged (88%), and minority 
(76%). The ITT sample included smaller proportions of students who were classified as special 
education (15%), needing additional accommodations (11%), limited English proficient (4%) or 
migrant (1%). Although the fixed effects impact analyses were not intended to generalize the 
findings to a larger population of schools, the students in the ITT sample assigned to treatment 
and control conditions appear to be a fairly diverse and representative sample of students from 
Title I schools geographically dispersed across the state of Louisiana. 


Levels of overall and differential attrition were within reasonable ranges. Overall attrition for the 
state LEAP assessment was 10% and differential attrition was 2.5%. Overall attrition for the 
GRADE and MRQ assessments were 20% while differential attrition was 3.5% and 2.6%, 
respectively. These levels of attrition result in low levels of potential bias for the study findings 
(What Works Clearinghouse, 2011). These levels of attrition also still enabled the study to 
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provide adequate statistical power for a minimum detectable effect size of 0.12 for the state 
reading assessment ({LEAP) and 0.13 for the standardized reading outcome assessment 
(GRADE). Additionally, the treatment and control groups maintained equivalence at all points in 
the study — post-randomization, in comparisons after pretest attrition, and in comparisons in after 
posttest attrition. There were never any significant demographic differences in the treatment and 
control groups. 


Along with low levels of attrition, the study design ensured high levels of random assignment 
integrity. There was a high degree of cooperation between study personnel and school liaisons 
and administrators. This helped largely avoid cases of crossover in treatment and control 
conditions, contamination of control students, and maintained original random assignment. 


What were the overall impacts of PRJ on student reading and other outcomes? This study 
of the PRJ program found evidence of a significant treatment effect on students' reading 
performance as measured by the overall GRADE reading assessment. Even after controlling for 
pretest scores, students in the treatment condition, on average, scored over three NCE points 
higher than students in the control condition (28.6 NCE vs. 25.3 NCE). The standardized effect 
size of the treatment was 0.27. 


Evidence of positive treatment effects were also found for the vocabulary and reading 
comprehension subscales on the GRADE assessment. Students in the treatment condition scored 
nearly 2 NCE points higher on the vocabulary subscale and nearly 4 NCE points higher on 
reading comprehension than students in the control group. The standardized effect size for 
GRADE vocabulary was 0.13 and 0.31 for reading comprehension. 


There was no evidence that assignment to the treatment condition had an effect on the state 
assessment scores. We found no evidence that students’ treatment assignment resulted in 
improvements in the overall ELA score or Reading subscore on the state LEAP assessment. 


Study hypotheses also proposed that assignment to the treatment condition would impact 
students' motivation to read. Analyses found no evidence that the treatment impacted students' 
motivation to read as measured by the MRQ for this study, either the total MRQ score or any of 
the four subscales. 


Were there differential impacts of PRJ on student subgroups? The results demonstrated 
significant differential impacts of the PRJ program by gender and grade for the overall GRADE 
reading and comprehension scores. The results suggest that for males and 7" graders especially, 
assignment to the PRJ treatment condition yielded positive and substantial gains in reading 
scores. Males in the treatment condition had significantly higher reading and comprehension 
scores (four to five NCE points) than their counterparts in the control condition, while the 
treatment condition had little effect on females’ GRADE reading outcomes. Similar results were 
found for 6" and 7" graders. For 6" graders, assignment to the treatment condition had a small, 
positive impact on the reading assessment. The differences between scores for treatment and 
control conditions were significantly larger for 7” graders. Seventh graders in the treatment 
condition scored nearly 2 NCE points higher than their counterparts in the control condition on 
the GRADE reading and comprehension assessments as well as on the 7LEAP reading subscale 
score. No significant differential impacts were found for the MRQ outcomes. In addition, no 
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significant differential impact findings were found for poverty, ethnicity, or baseline proficiency 
subgroups on any of the measured outcomes. 


The overall impact and differential subgroup impact results are encouraging. This study provides 
evidence that PRJ is effective at improving struggling 6" and 7" grade students’ reading 
performance on a standardized nationally norm referenced assessments after only one year of the 
intervention. The size of the effects, particularly for overall reading (0.27) and reading 
comprehension (0.31) were statistically significant as well as educationally meaningful (PRJ 
student scores increased 3 and 4 NCE points, respectively). The vocabulary effect size (0.13) 
was Statistically significant but relatively smaller (PRJ student scores increased 2 NCE points). 
Males appeared to benefit more from the PRJ program than females (males in PRJ increased 4-5 
NCE points relative to females) as well as older 7" grade students relative to 6" grade students 
(7" grade students in PRJ scores increased 2 NCE points relative to 6" grade students). These 
findings are particularly promising given the relatively small number of proven adolescent 
literacy programs for the struggling readers, who were the targets of this study. 
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Appendix A: Impact Analysis Methods 
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Treatment of Missing Data 


Multilevel modeling techniques provide model estimates taking into account certain types of 
missing data. In multilevel regression, the data are not actually imputed but the model is 
estimated making use of the incomplete data in a way that does not bias estimates under certain 
conditions. The estimation in HLM software (and more generally, REML estimation with other 
packages) is related to the missing data estimation for more general applications using the EM 
algorithm. There are a number of approaches to data analysis with incomplete data, including 
listwise deletion (in which all variables must be complete), pairwise deletion (in which only pairs 
of variables need to be complete), mean imputation (average value is substituted for the missing 
data), hot-deck imputation etc. Under certain restrictive conditions, when the data are missing at 
random (MAR) or missing completely at random (MCAR), missing data estimation with EM 
algorithm produces unbiased estimations of the complete data and better estimation than 
alternative approaches. 


There was the possibility of missing data for the demographic characteristics as well as the 
reading assessments. The demographic variables for students were compiled from the iLEAP 
data file provided by LDoE and were used to create subgroup analyses. There were no instances 
of missing data for gender, grade, and race/ethnicity. However, five students from the 1,226 
randomized students did not have information about their poverty status in 2010. Rather than 
exclude these students due to missing data, their poverty status was assigned based on previous 
years’ poverty classification which was also available in the data file. 


There was more substantial missing data for the reading assessments. The reading assessments 
were used as pretest predictor variables and as posttest outcome variables. For iLEAP, there was 
no missing data for the pretest as this was a condition for random assignment. Study students did 
have missing data for the posttest spring 2011 iLEAP assessment; 1,102 of the 1,226 students 
completed the 2011 iLEAP assessment (89.8%). Students who did not have posttest data were 
listwise deleted and not included in the final models. 


For the GRADE and MRQ, there were missing data for both the pretest and posttest. Nearly 90% 
of the sample completed the pretest assessments for the GRADE and MRQ. At the follow up 
posttest assessment, the completion rate was approximately 80%. All students with posttest 
assessments were included in the analyses. There were some cases of students (37 — 39 
depending on the assessment score used) where students had posttest but no pretest data. For 
these students, a dummy variable was set to one and their pretest score was set to zero, based on 
recommendations by Abt Associates. In no models did the dummy variable for missing pretest 
variable account for significant variation in the outcome variable, indicating the cases of missing 
pretest data did not represent serious bias. 
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Table A.1. Frequency and percent of complete data for pretest and posttest assessments 


Missing Data (Total Randomized = 1,226) 


Pretest Posttest 
N % Complete N % Complete 
iLEAP 1226 100.0% 1102 89.9% 
GRADE 1096 89.4% 983 80.2% 
MRQ 1097 89.5% 977 79.7% 


The completed consort diagrams for both the state test (LEAP), as well as the local evaluation 
baseline assessments (GRADE and MRQ) provide a graphical demonstration of the causes and 
timing of missing data. Tables A.2 — A.4 provide the consort diagrams for the three outcome 
variables. 
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POPULATION OF STUDENTS 


N= 3595 
ELIGIBLE STUDENTS INELIGIBLE STUDENTS 
n= 1437 n= 2158 
RANDOM analy 
TREATMENT GROUP CONTROL GROUP 
n= 720 n= 717 
Not Eligible for Study Post-RA Eligible for Treatment Eligible for Control Not Eligible for Study Post-RA 
n= 104 n= 616 n= 610 n= 107 
Attrition Post-RA | Attrition Post-RA 
n= 0 n= 0 
Sample at Baseline Sample at Baseline 
n= 616 n= 610 
Attrition at Follow-up Final Intent-to-Treat Final Intent-to-Treat Attrition at Follow-up 
n= 69 Analytic Sample at Analytic Sample at n= 55 
Attrition due to: Follow-up Follow-up Attrition due to: 
* Took LEAP 25 * Took LEAP 30 
* Took LAA2 12 n= 548 n= 554 * Took LAA2 10 
* No test scores 32 * No test scores 15 
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POPULATION OF STUDENTS 


N= 3595 
ELIGIBLE STUDENTS INELIGIBLE STUDENTS 
n= 1437 n= 2158 
RANDOM nih ag. 
TREATMENT GROUP CONTROL GROUP 
n= 720 n= 717 
Not Eligible for Study Post-RA Eligible for Treatment Eligible for Control Not Eligible for Study Post-RA 
n= 104 n= 616 n= 610 n= 107 
Attrition Post-RA Attrition Post-RA 
n= 84 n= 45 
Targeted Sample Targeted Sample 
at Baseline at Baseline 
n= 532 n= 565 
Attrition at Follow-up 2. Final Intent-to-Treat Attrition at Follow-up 
n= 131 Analytic Sample at Analytic Sample at n= 110 
Attrition due to: Follow-up Follow-up Attrition due to: 
* Moved to non-SR * Moved to non-SR 
school 48 school 55 
* Unable to locate, * Unable to locate, 
No test scores 23 n= 485 n= 500 No test scores 39 
* Promoted to 8" * Promoted to 8" 
grade or higher 11 grade or higher 6 
* Removed from PRJ * Removed from PRJ 
by administrator 31 by administrator 4 
* Parent opt out 13 * Parent opt out 3 
* Expelled 3 * Expelled 2 
* Other 2 * Other 1 
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POPULATION OF STUDENTS 
N= 3595 


be 


ELIGIBLE STUDENTS 
n= 1437 


INELIGIBLE STUDENTS 
n= 2158 


RANDOM tian. 


n= 


TREATMENT GROUP 


720 


Not Eligible for Study Post-RA 
n= 104 


Attrition Post-RA 
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CONTROL GROUP 
n= 717 


Eligible for Treatment 
n= 616 
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Eligible for Control 
n= 610 


! 


Not Eligible for Study Post-RA 
n= 107 


Attrition Post-RA 
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n= 84 n= 45 
Targeted Sample Targeted Sample 
at Baseline at Baseline 
n= 532 n= 565 


Attrition at Follow-up Final Intent-to-Treat Final Intent-to-Treat Attrition at Follow-up 

n= 132 Analytic Sample at Analytic Sample at n= 117 

Attrition due to: Follow-up Follow-up Attrition due to: 

* Moved to non-SR * Moved to non-SR 

school 48 school 55 
* Unable to locate, * Unable to locate, 

No test scores 24 n= 484 n= 493 No test scores 46 
* Promoted to 8" * Promoted to 8" 

grade or higher 11 grade or higher 6 
* Removed from PRJ * Removed from PRJ 

by administrator 31 by administrator 4 
* Parent opt out 13 * Parent opt out 3 
* Expelled 3 * Expelled 2 
* Other 2 * Other 1 
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Statistical Power 


The sample size necessary for this study was estimated based on a desired statistical power of 
.80, for an alpha level of p <.05, and the ability to detect an effect size of d= .10 (i.e., 0.10 
standard deviation). Minimum detectable effects (MDEs) are the smallest true impact one would 
expect to identify as statistically significant and are often scaled into standard deviation units or 
effect size units to improve comparability across studies or measures (Schochet, 2005). Power 
calculations were completed using Optimal Design for Multi-level and Longitudinal Design 
Version 2.0 software (Spybrook, Raudenbush, Congdon, & Martinez, 2009). Original estimates 
used the following parameters: the use of a pretest covariate (R°=.49), an intraclass correlation to 
accommodate for clustering of students within schools of 0.10 (James-Burdumy et al., 2009), 
and an effect size variability parameter of 0.00 to accommodate for modeling fixed effects at the 
school level, an estimated sample size of 1,200 students. We determined we could adequately 
detect an MDEs of .10 between the two conditions which is within our expected sample size 
estimate. 


Researchers anticipated retaining 80% of the target sample through the spring. In addition, Table 
A.5 presents the MDEs for subgroup analyses and demonstrates that the proposed study will 
have adequate power to detect an effect size of .16 or higher in subgroup analyses involving a 
sample as low as 25% of the total sample and will be able to detect an effect size of .23 - .26 in 
subsamples as low as 10% depending on the response/attrition rate. This table demonstrates the 
study would have adequate power to detect small effects (i.e., d =.14-.16) in the proposed 
subgroup analyses given that most, if not all, will involve subgroups of 25% or higher (e.g. 
grade, gender, years of program exposure — 50%, minority students — 50-75%). 


' The MDE estimates for subgroup analyses are based on an alpha=.05 and do not include corrections for the FWER which will 
be specified in later iterations of the analysis plan as we identify confirmatory hypotheses and specify models to test each 
hypothesis as well as preferred approaches to correcting for specific multiple hypotheses tested. 
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Table A.5. Statistical Power and Minimum Detectable Effect (MDE) Sizes, by Response Rate and 
Student Subgroup Configuration, full sample, 75 percent subsample, 50 percent 
subsample, 25 percent subsample, and 10% subsample 


Number 
of : 
Districts | Response sts ~ 75% 50% 25% 10% 
(K)/ Rate 8 Subsample | Subsample | Subsample | Subsample 
Sample 
Schools 
(J) 
ey me 900 600 300 120 
100% aes n=90 n=60 n=30 n=12 
J=10 MDE’=. 1 
0 ; MDE=.11 MDE=.12 MDE=.14 MDE=.23 
1080 820 540 280 100 
K=4 90% 
ar. n=108 n=82 n=54 n=28 n=10 
MDE=.11 | MDE=.11 MDE=.12 MDE=.15 MDE=.26 
960 720 480 240 100 
K=4 80% 
ip n=96 n=72 n=48 n=24 n=10 
MDE=.11 | MDE=.12 MDE=. 12 MDE=.16 MDE=.26 


Note. n= student n per school 


After posttest assessments were completed, power estimations were repeated using Optimal 
Design to determine the minimal detectable effect size based on the actual sample parameters as 
opposed to predicted sample parameters. For iLEAP, we estimated a fixed-effects model with 
approximately 1100 students and a pretest correlation of 0.49 to determine that we were powered 
to detect a MDES of 0.10. For GRADE, the MDES for a fixed-effects model with approximately 
980 students and a pretest correlation of 0.49 was 0.11. Attrition and missing data was not a 
substantial enough problem to seriously impact our ability to achieve a reasonable minimal 
detectable effect size. 


MDEs were calculated using Optimal Design Software for longitudinal and multilevel research version 2.0 assuming 80% 
power, alpha=.05, intraclass correlation=.10, and covariate (pretest) explained variance (R’= .49) at the student level for 


GRADE, an unblocked design (i.e., no blocking variables at the site (district) or school levels, and a balanced design (i.e., 50/50 
random assignment). 
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Figure A.1. Power Estimation based on posttest counts for iLEAP 
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Figure A.2 Power Estimation based on posttest counts for GRADE & MRQ 
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Results 
Model Specifications 


The principal goal of this evaluation is to investigate the effect of participating in the treatment 
(PRJ) versus business as usual for reading outcomes. 


Research Question: Does student-level assignment to the PRJ supplemental literacy program 
produce stronger effects on reading achievement for 6"" and 7" grade students significantly 
behind their peers (i.e., scoring “below basic” on the state assessment) in reading than 
assignment to supplemental non-reading elective services control condition? 


Two dependent variables will be used to assess the impacts of the program on reading fluency in 
ITT analyses: 


¢ GRADE (Group Reading Assessment and Diagnostic Evaluation) standard scores from 
the spring of each year on total reading, and four component scores and related subtests, 
including vocabulary (nouns, verbs, adjectives, adverbs subtests), reading comprehension 
(sentence comprehension, passage comprehension, and metacognition subtests) and oral 
language (listening comprehension subtest), and fluency (reading time, miscues, and 
comprehension subtests). 

¢ iLEAP (Integrated Louisiana Educational Assessment Program) norm-referenced English 
Language Arts and Reading Subscale test scores from the spring of each year. 


Research Question: Does student-level assignment to the PRJ supplemental literacy program 
produce stronger effects on student’s motivation to read for 6" and 7" grade students 
significantly behind their peers (i.e., scoring “below basic” on the state assessment) in reading 
than assignment to supplemental non-reading elective services control condition? 


Testing the Direct Effects of the Intervention on Student Outcomes 


The first two research questions involve testing the intention-to-treat (ITT) effects of the 
intervention on reading achievement and motivation. That is, what is the impact on student 
reading achievement outcomes of assignment to the PRJ supplemental intervention? This section 
of the analysis plan discusses the analytic approach to addressing this question. 


Methodological advances are still evolving regarding the best ways to model and test the impact 
of multi-site RCTs where, as in this study, the unit of randomization and the primary outcomes 
of interest are at the individual level. Increasing numbers of studies involve multilevel modeling 
techniques, sometimes referred to as Hierarchical Linear Modeling (HLM), as an analytic 
strategy for estimating the impact of multi-site RCTs. HLM or multilevel models are appropriate 
when, as in this study, outcomes are nested in a naturally occurring hierarchy of levels of 
analysis (e.g., when students are nested within schools). HLM in these situations is more 
appropriate than other regression techniques since such modeling properly accounts for the 
natural clustering and its associated correlation between students in the same school as well as 
correlations within students over time in a longitudinal study. 


Even within this analytic approach, the analyses for both individual-level and school-level 
outcomes can be thought of in the simplest form as a comparison of means for the treatment and 
control groups at the end of the intervention. One important advantage of an RCT design is that it 
greatly simplifies the data analysis task while providing an unbiased estimate of the program’s 
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impact on individual outcomes. Although this is a valid approach, randomization does not 
always ensure that groups are equal at the beginning of the intervention (pretest), and the 
reliability of the impact estimates can be substantially improved by the use of statistical control 
variables (other independent variables collected during the pretest). The most common multilevel 
model given the design of this study involves a two-level model’ that simultaneously accounts 
for student and school-level variation and clustering in outcomes and estimates the impact of the 
intervention as an individual-level effect. Researchers will estimate the impact of treatment 
assignment on the mean posttest achievement outcomes for students. Researchers will also 
include an individual-level covariate pretest score and school-level school 
membership/attendance dummy variable to assist in accounting for unexplained variance in 
outcomes and to improve the precision and corresponding power of the treatment effects 
estimates (Bloom, Bos, & Lee, 1999; Raudenbush, 1997). 


The fully specified level-1, or within-school model, will nest students within schools and assume 
within school j, the outcome Yij for participant 1 depends an individual-level covariate (pretest) 
and a treatment effect according to the regression model: 


Level 1 Model Yi = Boj a Bij*X1j a Boj* X2ij + Ti 


Where Y;; is the spring posttest achievement for student 7 in school /; Bo; is the mean outcome for 
school p Xi is a covariate (fall pretest) variable and X2j is a treatment contrast, i = 1...n; Bj; is 


the average covariate estimate on the outcome within school ;; ,; is the mean difference 
between outcomes of the experimental and control groups within school pp and rj is a individual- 
specific residual. 


In the level 2 model, the coefficients at level-1 become outcomes at level-2. Thus, the school- 
specific pretest covariate and treatment effect are fixed according to the model: 


Level 2 Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yo5*School 6+ yo6*School 7+ 


Model Yo7* School 8+ yog*School 9+ yo9*School 10 
Bij = Y¥i10 
Boj = Y20 


Where yoo is the grand mean outcome, and yo, through yo5 is the school specific mean 
differences; yo is the average pretest covariate effect, and y2. is the average treatment effect; no 
random error term is included as it is a fixed effects model. 


The parameter of central interest in the analysis is at level-1 in the model — the individual-level 
effect of assignment to the PRJ treatment (,;). This model is specified as a fixed-effects model 


? We will assess the value of modeling classroom-level variability and whether the three-level model explains appreciably more 
between-classroom variance and improves the precision of the treatment effect estimates. If it does not, we will fit a more 
parsimonious two-level model. 
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based on the assumptions of a conditional inference model (Hedges, 2009) in which there is not 
an intention to generalize effects beyond the current sample of purposively selected districts and 
schools to a larger population. 


For each outcome, models will be specified as part of an incremental modeling approach 
commonly used in multilevel modeling. A preliminary unconditional model is fit to partition the 
variance among the levels of analysis and as a basis for comparing the fit of subsequent models, 
which introduce student- and school-level predictors of posttest outcomes, including an ultimate 
model that adds an indicator of the treatment condition at the individual-level. 


Covariates 


We planned for the possibility that random assignment did not result in treatment and control 
conditions that were not equivalent and proposed including covariates in the impact analyses. 
The value of covariates in multilevel models used at multiple levels of analysis are to assist in 
accounting for unexplained variance in outcomes and to improve the precision and 
corresponding power of the treatment effects estimates. A list of proposed covariates can be 
found in Table A.6. 
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Table A.6. Proposed Independent Variables and Covariates, Data Sources, and Variable 


Independent 
Variables and 
Covariates: 


Data Source 


Variable Description 


Student grade level 


Archival student 
demographic profile 


Grade-level (i.e.,1= 6th and 0=7th) of 
each child in the study 


Student gender 


Archival student 
demographic profile 


Gender (male=1; female =0) of each 
child in the study 


Student race/ethnicity 


Archival student 
demographic profile 


Race/ethnicity of each child in the 
study (1=black; 0=non-black) 


Student SES Status 


Archival student 


Student Free- and reduced price lunch 


demographic profile status (i.e., no=0; yes=1) 
Student and School GRADE Standardized pre-test (fall) score 
Reading Status at 
entry into program 
Student Reading iLEAP Norm referenced test score from 


Status in Spring of 
Previous School Year 


previous spring 


Student motivation 
and engagement 


Student Survey 


MRQ total score and subscales 


Student exposure to SEDL Implementation | Exposure to program for each child in 

the program Study database the study. 

Teacher Qualifications | SEDL Implementation | Teacher Experience (1.e. total years 

Study database teaching), Education (i.e., highest 

degree attained), and Certifications. 

Teacher/Classroom SEDL Implementation | Specific and general implementation 

Fidelity of Study database fidelity scores 

Implementation 

Teacher/Classroom SEDL Implementation | Duration of program implementation 

Implementation Study database with fidelity (i.e., hours/days/weeks) 

duration 


Fortunately, the treatment and control conditions did not evidence any significant differences for 
either demographic characteristics or pretest reading scores. Analysis of variance provided 
evidence that the treatment and control groups were statistically similar when compared by 
iLEAP scores and demographic characteristics. The equivalence between treatment conditions 
remained even after pretest scores for the GRADE and MRQ assessment were collected. Thus, 
the proposed covariates (gender, grade, race/ethnicity, pretest) were used only as predictors (1.e., 
subgroup analyses to determine if these variables distinguish variation in impacts) rather than 
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covariates (i.e., to control for student or teacher factors) in the impact analyses. Comparisons of 
treatment and control condition demographic characteristics and pretest assessments are found in 
Table A.7. 
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Table A.7. Treatment and control condition equivalence 


Overall Treatment Condition Control Condition 
N % SD N % SD N % SD P value 

Male 1226 56.8% 0.5 617 56.6% 0.5 609 57.0% 0.5 0.880 
Race/Ethnicity 

Black 1226 70.6% 0.46 617 72.0% 0.45 609 69.1% 0.46 0.280 

White 1226 24.3% 0.43 617 22.9% 0.42 609 = 25.8% 0.44 0.230 

Other 1226 5.1% 0.22 617 5.2% 0.22 609 5.1% 0.22 0.940 
Grade 

6" 1226 48.0% 0.5 617 48.3% 0.5 609 47.6% 0.5 0.810 

his 1226 47.1% 0.5 617 47.2% 0.5 609 47.1% 0.5 0.990 

gt 1226 4.9% 0.22 617 45% 0.21 609 5.3% 0.22 0.560 


Special Education 1226 15.2% 0.36 617 15.4% 0.36 609 14.9% 0.36 0.820 


LEP 1226 3.6% 0.19 617 3.7% 0.19 609 3.4% 0.18 0.790 
Poverty status 1226 87.7% 0.33 617 88.3% 0.32 609 87.0% 0.34 0.490 
Migrant 1226 11% 0.1 617. 0.7% 0.08 609 15% 0.12 0.160 
Section 504 1226 10.5% 0.31 617. 11.0% 0.31 609 §=10.0% 0.3 0.570 
N mean SD N mean SD N mean SD P Value 


Pretest Equivalence 
of Analysis Sample 


iLEAP ELA 1102 246.8 36.3 548 246 37.1 554. 247.6 35.5 0.470 
GRADE Overall 944 83.2 89 455 83.3. 8.8 489 83 9.0 0.580 
GRADE 

Vocabulary 94] 87.5 10.7 454. 87.3 -—-:10.2 AT ORT AA 0.620 
GRADE 

Comprehension 944 82 9.0 455 82.4 9.0 489 81.7 9.0 0.300 
MRQ 940 28 04 455 28 04 485 28 04 0.980 
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Subgroup Analyses 


While the analyses presented above will address the overall ITT effects of the intervention, it is 
possible that particular subgroups of students will benefit more from PRJ than other subgroups. 
For instance, students with low pretest scores may benefit to a greater extent from PRJ than 
relatively higher-scoring students. By adding a cross-level interaction term to these basic models 
assessing the ITT effects of the treatment on student reading outcomes, researchers will be able 
to examine potential aptitude-by-treatment interactions or other forms of interactions of the 
treatment effect by varying student subgroups. 


These interaction effects between the treatment condition variable and particular baseline 
characteristics of students (e.g., grade, gender, ethnic minority status, initial reading proficiency, 
motivation) can assess whether the impacts were greater or smaller in magnitude under particular 
circumstances. For instance, testing the interaction effect between gender of the students and the 
treatment condition will address whether the impacts of the treatment on student outcomes were 
moderated by gender. Multilevel models are advantageous in such analyses due to the ability to 
accurately estimate the variance at the different levels of analysis. These analyses will be guided 
by a strategy for multiple comparisons testing in impact evaluations of educational interventions 
(Schochet, 2008) with particular consideration given to using confirmatory analytic approaches 
that control family-wise error rates for correlated test statistics, such as generalized Tukey and 
bootstrapping that will be specified in the analysis plan and informed by the technical assistance 
team. 


Several subgroup analyses were proposed to investigate for potential differences in the treatment 
effect between groups of students. 


Student-level by poverty subgroup — PRJ has statistically significant impacts on student reading 
achievement, specific student reading composite/subtests, and other student outcomes for 6th-7th 
grade students from low-income backgrounds compared to the outcomes of control students 
assigned to supplemental non-reading elective services. 


Student-level by ethnicity subgroup — PRJ has statistically significant effects on student reading 
achievement, specific student reading composite/subtests, and other student outcomes for 6-7th 
grade students from ethnic minority backgrounds compared to the outcomes of control students 
assigned to supplemental non-reading elective services. 


Student- level by grade subgroup — PRJ has statistically significant effects on student reading 
achievement, specific student reading composite/subtests, and other student outcomes in 6th 
grade compared to the outcomes of control students assigned to supplemental non-reading 
elective services. 

Student-level Time by Treatment subgroup — PRJ has statistically significant effects on student 
reading achievement, specific student reading composite/subtests, and other student outcomes at 
the end of the Ist year of student participation compared to the outcomes of control students 
assigned to supplemental non-reading elective services 


Student-level by gender subgroup — PRJ has statistically significant effects on student reading 
achievement, specific student reading composite/subtests, and other student outcomes for 6-7th 
grade students for males compared to the outcomes of control students assigned to supplemental 
non-reading elective services 
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Student-level by baseline proficiency in reading subgroup — PRJ has statistically significant 
effects on student reading achievement, specific student reading composite/subtests, and other 
student outcomes for 6-7th grade students who enter the program reading significantly worse 
than their peers compared to the outcomes of control students assigned to supplemental non- 
reading elective services. 


Student Motivation as Moderator/Mediator of Achievement — Students with higher motivation to 
read scores will show significantly greater improvement on student reading achievement and 
other specific composite and subtest reading outcomes. 


The HLM models resemble the ITT analyses but include an additional predictor variable that 
represents subgroup membership and an interaction term between the treatment condition and 
subgroup membership. 


Level 1 


Model Posttest; = Boj si Bij*X1j + Boj* X2ij + B3j*X3 ij + Baj*X2jj *X3 ij + Tj 


Where Y;; is the spring posttest achievement for student 7 in school /; Bo; is the mean outcome for 
school p Xi is a covariate (fall pretest) variable, X2; is a treatment contrast, § = 1...n, and X3;; 


is a subgroup contrast; {,; is the average covariate estimate on the outcome within school pp Bai is 
the mean difference between outcomes of the experimental and control groups within school p 

B3; 1s the mean difference between outcomes of the subgroups within school p and £,; is the 
interaction term between the treatment effect and subgroup membership within school p and rj is 
a individual-specific residual. 


Level 2 Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ 
Model Yoo*School 7+ yo7*School 8+ yos*School 9+ yoo*School 10 


Bi= Y10 
Boj = Y20 
B3j = 30 
Baj = Yao 


Where yoo is the grand mean outcome, and yo, through yo. is the school specific mean 
differences; y:) is the average pretest covariate effect, yx is the average treatment effect, ys. is 
the average subgroup effect, and y49 is the average interaction effect; no random error term is 
included as it is a fixed effects model. 
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Analytical Models 


The results for the models presented above are described in the body of the report. However, a 
complete listing of the overall impacts, model specifications, and analysis results can be found in 
Tables A.8- A.20. These tables provide the full HLM notation for the models as well as the 
regression estimates for all ITT and subgroup analyses. 
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Table A.8: i LEAP ELA Overall Score 


I. Treatment Impact Results 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B,,*Treatment + 1; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Mee ee ote a# Schaal Sie cy sekSchoal 9 vat Sokol 10 
Bi, = Yio 
Bo = 20 
Fixed Effects 

Level 

Student 0.000 
0787 
0.046 
0.588 
0.004 
0718 
0.589 
0.450 
0.282 
0.636 
0.000 
0252 
ee 

Fixed Effects 
Level Variance Components [Variance icc |sigma” | 
Student eveld | 259] 0.012] 126.854] 
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Table A.8: i LEAP ELA Overall Score 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Treatment + B3°*Poverty + By,*Poverty*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Fevel2 Model *Sohaol Sey School OX yia*School 10 
Bi, = Yio 
Bo, = 20 
B3; = 30 
Bai = Yao 

Fixed Effects 

Level 

Student 07000 
0.960 
0.085 
0.620 
0.004 
0-706 
0-703 
0.416 
0283 
0.619 
0.000 
0.602 
0.077 
0314 
ee Pe ee [ees eee! 

Fixed Effects 
Level Variance Components [Variance Sigma” |_| 
Student leveld | 258] 325.362] 
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Table A.8: i LEAP ELA Overall Score 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Treatment + B3;*Gender + By*Gender*Treatment + r; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Fe Model a at School 8a tSchool 9.e4,g*Schoul 10 
Bi, = Yi0 
Bo = 20 
B3) = 30 
Bai = Yao 
Fixed Effects 

Level 

Student 0000 
074i 
0.034 
0.539 
(0.005 
0.822 
0.433 
0340 
0208 
0.603 
0.000 
0.500 
0.000 
0.904 
aa a (ge (| | 

Fixed Effects 
Level Variance Components [Variance Sigma” {|_| 
Student levell | 276] 1296-446] | 
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Table A.8: i LEAP ELA Overall Score 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Treatment + B3°*Grade 6 + By,*Grade 6*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Fe Model a at School 8-iyactSchool 9e4,g8Schoul 10 
Bi, = Yi0 
Bo = Y20 
B3) = Y30 
Bai = Yao 

Fixed Effects 

Level 

Student 0-000 
0.883 
0.056 
0351 
0.003 
0.661 
0.634 
0338 
0336 
0739 
0.000 
0.403 
0.006 


Treatment*Grade 6, By; 0.035 1088 0.008 0.994 


Fixed Effects 
Level Variance Components __|Variance___|Sigma” | | 
Student 0.267 1311812y 
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Table A.8: i LEAP ELA Overall Score 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bp; + B,;*Pretest + B.,*Treatment + B3;*Black + B,*Black*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Fee Model at School 8-iyactSchool 9.e4,g*Schoul 10 
Bi, = Yi0 
Bo = Y20 
B3) = Y30 
Bai = Yao 

Fixed Effects 

Level 

Student 0-000 
0.902 
0.085 
0392 
0.004 
0.666 
0762 
0455 
0.293 
0738 
0.000 
0315 
0.920 


Treatment*Black, By, -2.083 1088 -0.434 0.664 


Fixed Effects 
Level Variance Components __|Variance___|Sigma” | | 
Student 0.257 1328764 | 
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Table A.8: i LEAP ELA Overall Score 


E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B,*Treatment + B3;*Pretest*Treatment + 1; 
Boi = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Fee del ist School: Sap tSchOOl Oe y4s*SchOOl 10 
Bi, = Yio 
Boj = Y20 
B3; = Y30 
Fixed Effects 
Level pte te 
Studer 


-0.064 0.061 1089 -1.051 
sss ees es SSS 


Fixed Effects 
Level Variance Components ___|Variance___|Sigma” | | 
Student 0.259 1326.725) Pd 
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0.000 
0.806 
0.051 
0.580 
0.004 
0.699 
0.621 
0.507 
0.277 
0.653 
0.000 
0.233 
0.293 
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Table A.9: i LEAP Reading Subscale Score 


I. Treatment Impact Results 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B,,*Treatment + 1; 
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Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pevel 2 Model» 5 AS choole LayytSehool 91 yetSehool 10 
Bi; = Y10 
Bo = 20 
Fixed Effects 
Level CE eee 
Student 
ee 
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Fixed Effects 


0.000 
0.454 
0.018 
0.296 
0.003 
0.207 
0.114 
0.885 
0.591 
0.638 
0.000 
0.927 
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Table A.9: i LEAP Reading Subscale Score 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Treatment + B3;*Poverty + By,*Poverty*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Poneh2: MOGED 7c, ¥Sehisal b-ysak SciGol 4 at Sehoe] 10 
Bi, = Yio 
Bo, = 20 
B3) = 30 
Bai = Yao 

Fixed Effects 

Level 

Student 0.000 
0.608 
0.077 
0317 
0.003 
0213 
0.162 
0916 
0.665 
0.602 
0.000 
o7m7 
0.119 


Treatment*Poverty, By; 3.418 9.449 1088 0.362 0.718 
ee ees re 


Fixed Effects 


Level Variance Components [Variance __[Sigma” |_| 
Student 0.162 2738335) 
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Table A.9: i LEAP Reading Subscale Score 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = B,; + B,;*Pretest + B,,*Treatment + B3;*Gender + By;*Gender*Treatment + r;; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


FEE MOE oj MSEHOGI RY nnag* SCHOO! OF Fat Sohool 10 
Bi, = Yio 
Bo = Y20 
B3; = Y30 
Bai = Yao 

Fixed Effects 

Level pe fieet —___t_Inpaet yf} SE Et ee 

Student 0-000 
0.437 
0018 
0.290 
0.004 
0197 
0.099 
0.954 
0.636 
0.632 
0.000 
0.793 
0aR4 


Treatment Male fy 508 0671 


Fixed Ettects 
Level Variance Components __[Variance___[Sigma”_— | | 
Student 0.151 2741513} Td 
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Table A.9: i LEAP Reading Subscale Score 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B,,*Treatment + B3°*Grade 6 + B,,*Grade 6*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ Yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Fee MO oj MSEHOGI BY nnag* SCHOO! OF ya Séhool 10 
Bi, = Yio 
Bo = Y20 
B3; = Y30 
Bai = Yao 

Fixed Effects 

Level 

Student 0-000 
0.462 
0.024 
0.286 
0.002 
0.190 
0.109 
0931 
0561 
0.668 
0.000 
0.123 
0.506 


Treatment*Grade 6, By; -13.990 6.326 1088 -2.212 0.027 


Fixed Effects 
Level Variance Components__[Variance___[Sigma”_— | | 
Student 0.154[ 2730665] | Cd 
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Table A.9: i LEAP Reading Subscale Score 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bp; + B,;*Pretest + B.,*Treatment + B3;°*Black + B,*Black*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


FEE MO oj MSEHGGI RY nae" SCHOO! OF ya Sohool 10 
Bi, = Yio 
Bo = Y20 
B3; = Y30 
Bai = Yao 

Fixed Effects 

Level pe tieet —__t_Inpae J SE Et oo 

Student 0-000 
0.652 
0073 
0306 
0.004 
0.602 
0942 
0811 
0410 
06 
0.000 
0.427 
0077 


Treatment" Black By satel tone] 503] 03 


Fixed eitects 
Level Variance Components__[Variance___[Sigma”— | | 
Student 0.162] 2707583} TC 
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Table A.9: i LEAP Reading Subscale Score 


E: Treatment*Pretest Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B,*Treatment + B3;*Pretest*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Fee MO oj MSEHOGI BY nag" SCHOO! OF eat Sobol 10 
Bi, = Yio 
Bo = Y20 
B3; = Y30 
Fixed Effects 
Level 
Student 


054] 1089 0.684 
a Ma Sa Hiss 


Fixed Effects 


Level Variance Components [Variance __[Sigma” |_| 


Student Level | 
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ois] 2743154) 


0.000 
0.433 
0.016 
0.307 
0.003 
0.195 
0.106 
0.915 
0.585 
0.615 
0.000 
0.916 
0.494 
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Table A.10: GRADE Overall Standard Score 


I. Treatment Impact Results 
Level 1 Model Posttest, = Bo; + B,;*Pretest + Bo;*Dorapr + B3; *Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pane? Model. yet school Seve? School Oe yast School 10 
Bi; = Y10 
Bo, = 20 
B3i = Y30 
Fixed Effects 

Level 

Student 0000 
0819 
0.003 
0.027 
0.059 
0418 
0342 
0513 
0134 
0.629 
0.000 
0.118 
0.000 

Fixed Effects 
Level Variance Components [Variance ficc_____|sigma” | 
Student level! CT C863] 0.020] 5.217 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.10: GRADE Overall Standard Score 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,*Poverty + B;*Poverty*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel? Model. 5-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 07000 
0.940 
0.002 
0.026 
0.051 
oni 
0.430 
0518 
0.107 
0.566 
0.000 
0.120 
0.007 
Poverty. py i 08a ———« CT ——i 
0416 

Fixed Effects 
Level Variance Components [Variance __[Sigma”__— |_| 


Student 0.364 S1i26gf CT 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.10: GRADE Overall Standard Score 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Doprapr + B3; *Treatment + B,**Male + B;,*Male*Treatment + r; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 5-# School Biewigr School 94-7 y*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 07000 
0.855 
0.002 
0.026 
0.067 
0.433 
0371 
0.479 
0116 
0.619 
0.000 
0.132 
0.093 
001 
0.016 

Fixed Effects 
Level Variance Components [Variance Sigma” {|_| 
Student level! CT 866] 50.956] CT 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.10: GRADE Overall Standard Score 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest,; = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,**Grade 6 + B;;*Grade 6*Treatment + 1; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 1-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0.000 
074d 
0.001 
0011 
0.053 
osm 
0-282 
0.496 
0.059 
0561 
0.000 
0.101 
0.000 
0.000 
0014 

Fixed Effects 
Level Variance Components [Variance _[Sigma” |_| 
Student evel! E370 500624) 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.10: GRADE Overall Standard Score 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dopapr + B3; *Treatment + B,**Black + B;;*Black*Treatment + r, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model. "5 #Schoel Biewggr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 07000 
0.962 
0.002 
0.024 
0.055 
0.370 
0572 
0512 
0137 
0.816 
0.000 
0.099 
0.000 
0-750 
0.087 

Fixed Effects 
Level Variance Components [Variance Sigma” |_| 
Student evel! E864 SIS| CT 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dopapr + B3; *Treatment + By*Pretest*Treatment + 1; 
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Bo; = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


PANEL Mode: og AS chOOI BE AGES CHOOL 9 Loi tSCHOOL10 
Bi; = Y10 
Boj = 20 
B3) = 30 
Baj = Y40 
Fixed Effects 
Level Efea ———S~s~Cipet@) [SSCP TY 
Student nereepi By 8107] TOR aL 
School Ly, —*(| 07] ___o-naol 960 0.228 
School 3, wes 48085] —= 
School 4. yoy 2608] ————«L SCT 
School 5, yas A057] 901.859 
School ys] 0S] ——«| CORT 
School 7, ve sf 09> —_09as| 969 0.965 
School 8 yoy = 08] ———908|__—-06T 
School 9, yw a —99[ 99 =a 
School 10, yy —~4| 0si2| 106i 960[ ana 
Pretest py, =i 800] if _—aS T 
Dorape'By | =1852] 7] 969] 1.560 
Frreaiment, fy | 2.366 o4i] ___969[ 3.131 
Frreatment*Pretest.By | 0.034] 0.053] 969[ 0.639 
a ee ee 
i Effects 
ot 


Level —— —— 
Student Zi a 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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0.000 
0.823 
0.002 
0.029 
0.064 
0.408 
0.335 
0.540 
0.146 
0.630 
0.000 
0.119 
0.000 
0.523 
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Table A.11: GRADE Overall NCE 


I. Treatment Impact Results 
Level 1 Model Posttest, = Bo; + B,;*Pretest + Bo;*Dorapr + B3; *Treatment + rj, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Panel? Model. ya tschool 8eyst School Se yagtSchool 10 
Bi; = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0-000 
0-805 
0.002 
0.022 
0.051 
0383 
0.283 
0382 
0.076 
0.661 
0.000 
0.203 
0.000 

Fixed Effects 
Level Variance Components [Variance ficc____|sigma” | 
Student level! CT 872 0.020] 95.458] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.11: GRADE Overall NCE 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,;**Poverty + B;*Poverty*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 
Yo7* School 8+ yYog*School 9+ yo9*School 10 


Level 2 Model 


Bi; = Y10 
Boj = Y20 
B3; = Y30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 07000 
0.940 
0.001 
0.020 
0.042 
0.376 
0371 
0.588 
0.057 
0.590 
Pretest By SSCS ~«é RY ~C~*«SBGT_—_—_0.000 
0.205 
0.005 
0.393 
0.388 

Fixed Effects 
Level Variance Components [Variance __[Sigma”__— |_| 


Student 0.374 95.212] | 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.11: GRADE Overall NCE 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,**Male + B;,*Male*Treatment + r, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 5-# School Biewigr School 94-7 y*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0-000 
0.842 
0.002 
0.021 
0.057 
0304 
0311 
0.546 
0.066 
0.652 
0.000 
0.219 
0.078 
0.022 
0018 

Fixed Effects 
Level Variance Components [Variance [Sigma” {|_| 
Student evell E875] 95.027] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.11: GRADE Overall NCE 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Doprapr + B3; *Treatment + B,**Grade 6 + B;°*Grade 6*Treatment + rj, 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 1-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0-000 
0.729 
0.001 
0.009 
0.086 
047i 
0230 
0.564 
0.054 
0.590 
0.000 
0.000 
0.000 
0077 

Fixed Effects 
Level Variance Components [Variance _[Sigma” |_| 
Student evell E79 94.370) ST 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.11: GRADE Overall NCE 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dopapr + B3; *Treatment + B,**Black + B;;*Black*Treatment + r, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel 2 Model. "5 #Schoel Biewggr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0.000 
om 
0.001 
0.019 
0.087 
0.328 
0528 
0.580 
0.079 
087 
0.000 
0.172 
0.000 
0.804 
0.079 

Fixed Effects 
Level Variance Components [Variance [Sigma”_—— |_| 
Student eevelt BJ 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.11: GRADE Overall NCE 


E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + Bo,*Dorapr + B3; *Treatment + B,*Pretest*Treatment + 1; 
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Bo; = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


PANEL Mode: og AS chOOI BE AGES CHOOL 9 Loi tSCHOOL10 
Bij = Yio 
Boj = Y20 
B3j = 30 
Baj = Y4o 

Fixed Effects 

Level Efea ———S~S~sSCipet@) |S dC Td 

Student Inept fy 28 TOD] WY“ 
School Ly) (| 0290[ 1202] 960[ 0.248 
School 3, wes 85a] iC 
School 4. yoy «Ti ———~«Y—~—~=« 
School 5, yw +P 289] 1 306f 969d 
School 6. ys ORT] 208 ——« | COS 
School 7, ve | aa 290[ 969 1.085 
School 8 yoy «| Ta] saa ——__—808| =a 
School 9, yw i 808i] iTS 
School 10, yy = 0] saa ———«] CO 
Pretesi py, =i HYCO] =i iS SG 
Dorave'By | 2.057] 6201969] 1.269 
Frreaiment, fy | _3274[___0629[ 960 3.209 
a a A A 

a ee eee! 
i eitects 
Level Variance Components [Variance Sigma” |_| 
Sie a 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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0.000 
0.809 
0.002 
0.024 
0.056 
0.371 
0.274 
0.621 
0.087 
0.660 
0.000 
0.205 
0.000 
0.414 


A-42 


Louisiana Striving Readers: Final Evaluation Report | Appendix A 


Table A.12: GRADE Vocabulary Scale Score 


I. Treatment Impact Results 


Level 1 Model Posttest,, = Bo; + B,;*Pretest + Bo;*Dorapg + B3; *Treatment + 1, 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Panel? Model. ya tschool 8eyst School Se yagtSchool 10 
Bi; = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0000 
0719 
0.290 
0410 
0227 
oi 
0516 
0976 
0219 
0725 
0.000 
0.050 
0.021 

Fixed Effects 
Level Variance Components [Variance _ficc___|sigma” | 
Student Level! CT 228] 0.006] 75.825] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.12: GRADE Vocabulary Scale Score 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,;*Poverty + B;*Poverty*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel? Model. 5-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
Ori 
0.267 
0.401 
0218 
0.108 
0.433 
0.925 
0.189 
0.688 
0.000 
0.049 
0551 
0227 
0762 

Fixed Effects 
Level Variance Components [Variance [Sigma”_ |_| 
Student level! T0230] 75.831] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.12: GRADE Vocabulary Scale Score 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Doprapr + B3; *Treatment + B,**Male + B;,*Male*Treatment + r;, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 5-# School Biewigr School 94-7 y*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.697 
0.286 
0.398 
0727 
0.108 
0479 
0.881 
0225 
0.726 
0.000 
0.046 
057 
0.509 
0.183 

Fixed Effects 
Level Variance Components [Variance __[Sigma”_— {|_| 
Student level) E228] 75.831) 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.12: GRADE Vocabulary Scale Score 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Doprapr + B3; *Treatment + B,**Grade 6 + B;;°*Grade 6*Treatment + rj, 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 1-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.792 
0220 
0281 
0.225 
0.138 
057 
0879 
0177 
0.648 
0.000 
0.058 
0.030 
0.020 
0427 

Fixed Effects 
Level Variance Components [Variance [Sigma”_— {|_| 
Student level) T0232] 75.456] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.12: GRADE Vocabulary Scale Score 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dopapr + B3; *Treatment + B,*Black + B;;*Black*Treatment + r, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 
Yo7* School 8+ yYog*School 9+ yo9*School 10 


Level 2 Model 


Bi; = Y10 
Boj = Y20 
B3; = Y30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 0-000 
0.426 
0233 
0.380 
0731 
0.067 
0216 
0.892 
0.260 
0.951 
0.000 
0.043 
0.034 
0.456 
0341 

Fixed Effects 
Level Variance Components [Variance __[Sigma” |_| 
Student evell | 230] 75.673] | 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.12: GRADE Vocabulary Scale Score 


E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dopapr + B3; *Treatment + B,*Pretest*Treatment + 1; 
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Bo; = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


PAVED Model. ye School Bieaget School oxy gt School 10 
Bi; = Y10 
Boj = Y20 
B3; = Y30 
Baj = Y40 

Fixed Effects 

Level Efea ———S~S~sSCipaet@) | SSCP TY 

Student nereepi By 80216] 085 va 100809 
School Ly —*4| —082[ oa] 900] -0366 
School 3, wes 88am] ——88f——= 
School 4, yoy 238] asa ———808|——i ST 
School 5, yw | TS] _—969[ 
School Yosh T08[ ora] 900[ 1580 
School 7, ve OTSA] =a C—O 
School 8 yoy =] ——«RY COTS 
School 9, yw sf RSS] ———= WL —* 
School 10, yy —~4| 0426] 282 —«8|—i0 880 
Pretesi py ———=d 0086] ———8af 12.929 
DeaaneBy 236 1392[ a0] 1.965 
resent F0[“aseol sea| 30 
am wal —— 00s ——ve9] —a 

a ae ee 
i. Effects 
Level Ea 
Student level! T0228] 75.860} 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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0.000 
0.714 
0.303 
0.395 
0.224 
0.114 
0.513 
0.887 
0.210 
0.742 
0.000 
0.050 
0.019 
0.457 
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Table A.13: GRADE Vocabulary NCE 


I. Treatment Impact Results 
Level 1 Model Posttest,; = Bo; + B,;*Pretest + Bo;*Dorapr + B3; *Treatment + 1; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pane? Model. yet school Seve? School Oe yast School 10 
Bi = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0.000 
0.656 
0337 
0371 
0279 
0.086 
0.499 
0.804 
0267 
0821 
0.000 
0.205 
240i, 0017 

Fixed Effects 
Level Variance Components [Variance ficc_____|sigma” | 
Student Level! CT 235] 0.006] 142 48] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.13: GRADE Vocabulary NCE 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,*Poverty + B;*Poverty*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 
Yo7* School 8+ yYog*School 9+ yo9*School 10 


Level 2 Model 


Bi; = Y10 
Boj = Y20 
B3; = Y30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 07000 
0557 
0311 
0363 
0.268 
0.084 
0.420 
0.902 
0233 
0.780 
0.000 
0.201 
0.462 
0.266 
0.854 

Fixed Effects 
Level Variance Components [Variance [Sigma”_— {|_| 
Student levell E2386] ago} | 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.13: GRADE Vocabulary NCE 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Doprapr + B3; *Treatment + B,**Male + B;,*Male*Treatment +r; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 5-# School Biewigr School 94-7 y*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.632 
0333 
0.359 
0279 
0.083 
0.460 
0.857 
0274 
0.823 
0.000 
0.195 
0-58 
0.495 
0.157 

Fixed Effects 
Level Variance Components [Variance [Sigma”_—— {|_| 
Student levell E2385] 14391 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 


SEDL | Research & Evaluation A-51 


Louisiana Striving Readers: Final Evaluation Report | Appendix A 


Table A.13: GRADE Vocabulary NCE 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,**Grade 6 + B;°*Grade 6*Treatment + 1, 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 1-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0-728 
0.255 
0.246 
0279 
0.108 
0.560 
0.851 
0214 
0739 
0.000 
0.252 
0.040 
0.022 
0587 

Fixed Effects 
Level Variance Components [Variance _[Sigma”_— {|_| 
Student evel! T0239] 625} 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.13: GRADE Vocabulary NCE 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dopapr + B3; *Treatment + B,**Black + B;;*Black*Treatment + r, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel 2 Model. "5 #Schoel Biewggr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
oni 
0299 
0346 
0-282 
0.055 
0233 
0874 
0.306 
0.887 
0.000 
0.182 
0.030 
0547 
0335 

Fixed Effects 
Level Variance Components [Variance [Sigma”__— |_| 


Student 0.236 142.216f | 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.13: GRADE Vocabulary NCE 


E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dopapr + B3; *Treatment + B,*Pretest*Treatment + 1; 
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Bo; = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


PANEL Mode: og AS chOOI BE AGES CHOOL 9 Loi tSCHOOL10 
Bij = Yio 
Boj = Y20 
B3j = 30 
Baj = Y4o 

Fixed Effects 

Level Efea ————S~s:Cipet@) | Sd 

Student JinereepiPy TT] 50] 0s 
School Ly, *4 eT] aoa ———~—«| SCO 
School 3, wes OTe] 890 
School 4. yyy] 83 ——«L ~~ 
School 5, yas T8619] 969 1.089 
School os] sao ————=|—— TIO 
School 7, ye 1068175] ___9o0 0.678 
School 8 yoy «| 0297] rea] —__—«8ea| = O10 
School 9, yw sf 220i 0 895{ 8S 
School 10, yy +4 oseif__r7of 900 0.204 
Pretesi py, =i ara 0086] 969 13.230 
DeaanBy 20 _.979f ooo) zs 
reament [ial soa] 2 
po a= OS ie = 

a ee eee 
____ Effects 
Level —————— 
Student leveld T2385] 4g9zp | 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 


SEDL | Research & Evaluation 


0.000 
0.650 
0.352 
0.355 
0.276 
0.088 
0.498 
0.866 
0.257 
0.838 
0.000 
0.206 
0.016 
0.481 
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Table A.14: GRADE Comprehension Standard Score 


I. Treatment Impact Results 
Level 1 Model Posttest, = Bo; + B,;*Pretest + Bo;*Dorapr + B3; *Treatment + 1; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pane? Model. yet school Seve? School Oe yast School 10 
Bi = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0000 
0263 
0.000 
0.036 
0.060 
0.966 
0129 
oii 
0221 
0262 
0.000 
0.243 
sani. 0.000 

Fixed Effects 
Level Variance Components [Variance ficC_____|sigma” | 
Student Level! CT 268] 0.022] 64.529] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.14: GRADE Comprehension Standard Score 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,*Poverty + B;*Poverty*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel? Model. 5-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
School try, iY 0807] of sent oil 0360 
0.000 
0.033 
0.087 
School 6. ys 00] safe] ——iR 
0.189 
om 
0168 
0214 
0.000 
0.250 
0.001 
0.425 
0174 

Fixed Effects 
Level Variance Components [Variance _[Sigma” |_| 
Student evell T2784} CT 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.14: GRADE Comprehension Standard Score 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Doprapr + B3; *Treatment + B,**Male + B;,*Male*Treatment + r;, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 
Yo7* School 8+ yYog*School 9+ yo9*School 10 


Level 2 Model 


Bi; = Y10 
Boj = Y20 
B3; = Y30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 0000 
0279 
0.000 
0.036 
0.070 
0.930 
0138 
0.387 
0.186 
0.250 
0.000 
0,282 
0.060 
0.005 
0.020 

Fixed Effects 
Level Variance Components [Variance [Sigma”_ {|_| 
Student leveld | 272427] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.14: GRADE Comprehension Standard Score 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,**Grade 6 + B;;°*Grade 6*Treatment + 1, 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 
Yo7*School 8+ yYog*School 9+ yo9*School 10 


Level 2 Model 


Bi; = Y10 
Boj = Y20 
B3; = Y30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 0000 
0221 
0.000 
0014 
0.051 
0.825 
0.096 
0.405 
0.162 
0.221 
0.000 
Ee a EE 0.189 
0.000 
a LC 0.000 
3231023 0.001 

Fixed eitects 
Level Variance Components [Variance [Sigma” |_| 
Student eveld CE 279 wf 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.14: GRADE Comprehension Standard Score 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,**Black + B;;*Black*Treatment + r, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel 2 Model. "5 #Schoel Biewggr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.410 
0.000 
0.031 
0.055 
0.926 
0316 
0.410 
0228 
0.428 
0.000 
0.207 
0.000 
0.832 
0.068 

Fixed Effects 
Level Variance Components [Variance [Sigma”_ |_| 
Student evel’ OPC 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.14: GRADE Comprehension Standard Score 


E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dopapr + B3; *Treatment + B,*Pretest*Treatment + 1; 
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Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


PANEL Mode: og AS chOOI BE AGES CHOOL 9 Loi tSCHOOL10 
Bij = Yio 
Boj = Y20 
B3j = 30 
Baj = Y4o 
Fixed Effects 

Level Efea ——S~S~sSCimpet@) |S dC Td 

Student YInereept fy BOT 8010 
School try, «dt =| 09a] 969 1.086 
School 3, wes ei] ——=af C= 
School 4, yoy |i] ——~=«tL—~C~=«tCG 
School 5, yas RYT] 969i 
School 6. ys 0.008] 099i 900 0.009 
School 7, ve YA] 050| =o ——*a 
School 8 yoy = 0820] ia C=C 
School 9, yw =i as] ——«8a——* 
School 10, yy | 288] aso ———908[_——-08 
Pretesi py, =i aT] ——_oni] 969 10.889 
Dorape'By | 525] 2.330) 969] 1.147 
Frreatment, fy +f 74]___ ost] ____960[ 3.299 
a a a a 

a ee eee 
_____. Effects 
Level Variance Components [Variance _[Sigma” {|_| 
Student leveld | 2704306] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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0.000 
0.278 
0.000 
0.042 
0.079 
0.993 
0.110 
0.489 
0.283 
0.278 
0.000 
0.252 
0.000 
0.037 
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Table A.15: GRADE Comprehension NCE 


I. Treatment Impact Results 
Level 1 Model Posttest, = Bo; + B,;*Pretest + Bo;*Dorapr + B3; *Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pane? Model. yet school Seve? School Oe yast School 10 
Bi = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0000 
0.246 
0.000 
0.023 
0.056 
0.982 
0.074 
0338 
0.164 
0335 
0.000 
0.256 
5527] 0.000 

Fixed Effects 
Level Variance Components [Variance ficc___|sigma” | 
Student level! CT 293] 0.023] 115.620] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.15: GRADE Comprehension NCE 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dorapr + B3; *Treatment + B,;*Poverty + B;*Poverty*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel? Model. 5-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 07000 
0.335 
0.000 
0.021 
0.044 
0970 
0112 
0.342 
0.124 
0.282 
0.000 
0.262 
0.001 
0.427 
0214 

Fixed Effects 
Level Variance Components [Variance __[Sigma”__— |_| 


Student 0.295 soo] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.15: GRADE Comprehension NCE 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dopapr + B3; *Treatment + B,**Male + B;,*Male*Treatment + r, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 5-# School Biewigr School 94-7 y*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.263 
0.000 
0.022 
0.064 
0.991 
0.081 
0314 
0141 
0.325 
0.000 
0.285 
0.086 
0.013 
0.022 

Fixed Effects 
Level Variance Components [Variance __[Sigma”__— |_| 


Student 0.296 is.0so] CT 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.15: GRADE Comprehension NCE 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Doprapr + B3; *Treatment + B,**Grade 6 + B;;°*Grade 6*Treatment + rj, 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model." 1-#Schoel Biewigr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 07000 
0212 
0.000 
0.008 
008 
0.879 
0.053 
0.330 
0.116 
0.294 
0.000 
0.205 
0.000 
0.000 
0.002 

Fixed Effects 
Level Variance Components [Variance [Sigma”_— |_| 
Student level) T0803 3.841] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.15: GRADE Comprehension NCE 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dopapr + B3; *Treatment + B,**Black + B;;*Black*Treatment + r, 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel 2 Model. "5 #Schoel Biewggr School 94-7 g*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.406 
0.000 
0.019 
0.052 
0.862 
0724 
0.335 
0172 
0.536 
0.000 
0.218 
0.000 
0.920 
0.081 

Fixed Effects 
Level Variance Components [Variance Sigma” |_| 
Student levell T0295] 05.288} 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.15: GRADE Comprehension NCE 


E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dopapr + B3; *Treatment + By*Pretest*Treatment + 1; 
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Boj = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


PAVED Model. ye School Bieaget School oxy gt School 10 
Bi; = Y10 
Boj = 20 
B3) = 30 
Baj = Y40 

Fixed Effects 

Level Efea ——S~sSCimpet®) |S dC Td 

Student Inereepe fy 21S Sa] 8 
School Ly) | Soi] sof ——«8] CS 
School 3, we dP SOS] _——=8[—= 
School 4. yoy 3888] TOD——=| = 
School 5, yas Oi] = 
School 6. ys ——= 00TH] isa ———~—« BL —~—~OOST 
School 7, ve 268] _—_—99[——= ST 
School 8 uy = |_| ——~C~=«t LCG 
School 9, yyw | 22701800] 969 1286 
School 10, yy 2] isa] ——~—«T—C~C« 
Pretest py, =i aR] ———0ni| 969 11.810 
Dorape'sBy | 2.001178 969] 1.124 
ee a) oo) 
a a a 

aay ae eee 
i Effects 
Level Variance Components [Variance Sigma” |_| 
Student levell T2944] 


“Dorapr - Indicator for missing pretest data (0 = non-missing, | = missing). 
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0.000 
0.256 
0.000 
0.026 
0.070 
0.954 
0.064 
0.398 
0.208 
0.351 
0.000 
0.261 
0.000 
0.092 
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Table A.16: Motivation to Read: Overall Motivation 


I. Treatment Impact Results 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.;*Dypo + B3, *Treatment + 1;; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Panel? Model. yet school Says? School Se yastSehool 10 
Bi; = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0-000 
0.000 
0.462 
0.862 
0.868 
0.008 
0.004 
082 
0.088 
0.004 
0.000 
0.923 
0227 

Fixed Effects 
Level Variance Components [Variance ficc_____|sigma” | 
Student level! | 0823] 0.064] 0.130] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.16: Motivation to Read: Overall Motivation 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + B3, *Treatment + B,,*Poverty + B;;*Poverty*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel? Model." 5_ A School Biewigr School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.000 
0.432 
School dys ———~dY 00] oat ——90a|—0.t9a——OR a 
0.857 
0.008 
0.007 
0.864 
0.037 
0.004 
0.000 
0.950 
0207 
0.087 
Freaiment-Povery By [| _-006| 0067] 9a] oma] 0373 

Fixed Effects 
Level Variance Components [Variance [Sigma”_— |_| 
Student evel! 39297 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.16: Motivation to Read: Overall Motivation 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + 63, *Treatment + By *Male + B;°*Male*Treatment + 1; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ y¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model." 5_ A Schoel Biewigt School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.000 
0.462 
0.859 
0875 
0.008 
0.005 
0.829 
0.088 
0.004 
0.000 
0.914 
0205 
0.643 
0537 

Fixed Effects 
Level Variance Components [Variance [Sigma”_— |_| 
Student evell T0822 0.80) 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.16: Motivation to Read: Overall Motivation 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + B3, *Treatment + B,,*Grade 6 + B;;*Grade 6*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model. "5A School Biewigt School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.000 
0523 
0.984 
0-880 
0.009 
0.005 
0.866 
0.058 
0.005 
0.000 
0,899 
0.073 
0.029 
0.180 

Fixed Effects 
Level Variance Components [Variance __[Sigma”__— |_| 


Student 0.325 0.129) 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.16: Motivation to Read: Overall Motivation 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dyypo + B3, *Treatment + By,*Black + B;;*Black*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model. "5A School Biewigr School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 07000 
0.006 
0326 
0742 
0919 
0.060 
0.416 
0.808 
0.084 
0.099 
0.000 
0.984 
0.993 
0.000 
0.540 

Fixed Effects 
Level Variance Components [Variance _[Sigma”_— |_| 
Student levell THOT 2G} 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.16: Motivation to Read: Overall Motivation 
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E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dypo + B3; *Treatment + By; 


*Pretest*Treatment + 1; 


Level 2 Model 
ES School 8+ Yox*School 9+ Yoo*School 10 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Bi, = Yio 
Boj = Y20 
B3; = Y30 
Baj = Y40 
Fixed Effects 

Level 

Student 0.000 
0.000 
0.464 
0873 
0.870 
0.008 
0.005 
0.838 
0.051 
0.005 
0.000 
0.923 
0.226 
0875 
ea ae (er | (|) 

Fixed Effects 
Level Variance Components [Variance Sigma” {|_| 
Student eveld E822 30 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.17: Motivation to Read: Intrinsic Motivation 


I. Treatment Impact Results 
Level 1 Model Posttest,; = Bo; + B,;*Pretest + By;*Dypo + B3, *Treatment + 1;; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Panel? Model. yet school Says? School Se yastSehool 10 
Bi; = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0000 
0.000 
0.890 
087 
0.803 
0.074 
0.006 
0761 
0131 
0.001 
0.000 
0208 

Fixed Effects 
Level Variance Components [Variance ficc_____|sigma” | 
Student level! OT 20.054] 34 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.17: Motivation to Read: Intrinsic Motivation 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bp; + B,;*Pretest + B2,*Dypo + 63, *Treatment + B,,*Poverty + B;;*Poverty*Treatment + r;; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model." 5_ A School Biewigr School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.000 
080 
0.888 
orm 
0.074 
0.010 
0779 
0.105 
0.001 
0.000 
0.859 
0.496 
0178 
0773 

Fixed Effects 
Level Variance Components [Variance __[Sigma”_ |_| 
Student levell T2293} 


“Duro = Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.17: Motivation to Read: Intrinsic Motivation 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypq + B3, *Treatment + By*Male + B;°*Male*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model." 5_ A Schoel Biewigt School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.000 
0.892 
0.850 
0847 
0.069 
0.006 
0.752 
0.116 
0.001 
0.000 
0.799 
0.189 
0.130 
0.461 

Fixed Effects 
Level Variance Components [Variance [Sigma”_ |_| 
Student levell TBA CT 


“Duro = Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.17: Motivation to Read: Intrinsic Motivation 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + B3, *Treatment + B,,*Grade 6 + B;;*Grade 6*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model. "5A School Biewigt School 94-7 3*School 10 
Bi; = Y10 
Boj = Y20 
B3; = Y30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.000 
0.939 
0775 
0819 
962] ____-1.728| 0.084 
0.007 
0-769 
School 9, yw | 0090062] 96a __-1.4sa 0.146 
0.054 0.002 
0.031 0.000 
0.770 
Frreatment, fy | __ 006 0084 0.077 
0.055 0.096 
0.087 0.176 

Fixed Effects 
Level Variance Components [Variance Sigma” {|_| 
Student level) TBA CT 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.17: Motivation to Read: Intrinsic Motivation 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dyypq + B3, *Treatment + By,*Black + B;;*Black*Treatment + 1; 


Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model. "5A School Biewigr School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 07000 
0.003 
0758 
0.955 
0.862 
0275 
0331 
0-728 
0210 
0.030 
0.000 
0.899 
0534 
0.001 
0.901 

Fixed Effects 
Level Variance Components [Variance __[Sigma”__— |_| 


Student 0.230 0.132) 00 


“Duro = Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.17: Motivation to Read: Intrinsic Motivation 
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E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B.,*Dypo + B3; *Treatment + By; 


*Pretest*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model." 5-# School Biewggr School 94-7 g*School 10 
Bi, = Yio 
Bo = 20 
B3) = 30 
Baj = Y40 

Fixed Effects 

Level 

Student 0.000 
0.000 
0891 
0.868 
0.804 
0.074 
0.006 
0761 
0.133 
0.002 
0.000 
0.844 
0247 
0924 
Pe 

Fixed Effects 
Level Variance Components [Variance Sigma” |_| 
Student evel! TG BA 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.18: Motivation to Read: Extrinsic Motivation 


I. Treatment Impact Results 
Level 1 Model Posttest, = Bo; + B,;*Pretest + By;*Dypo + B3, *Treatment + 1;; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Panel? Model. yet school Says? School Se yastSehool 10 
Bi; = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0000 
0.000 
0291 
0.639 
0399 
0.001 
0.001 
0.887 
0.08 
0.055 
0.000 
0.928 
0.892 

Fixed Effects 
Level Variance Components [Variance __ficc_____|sigma’” | 
Student level! OT 0230 0.043] 0.238] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.18: Motivation to Read: Extrinsic Motivation 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + 63, *Treatment + B,,*Poverty + B;;*Poverty*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model." 5_ A School Biewigr School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 0000 
0.001 
0272 
0.624 
0.401 
School 6 ys 0] efi] 0.001 
0.002 
0.909 
0.032 
0.052 
0.000 
0.161 
0.028 
0.140 

Fixed Effects 
Level Variance Components [Variance [Sigma”_— |_| 
Student evell T0252] 0.237] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.18: Motivation to Read: Extrinsic Motivation 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + 63, *Treatment + By*Male + B;°*Male*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model." 5_ A Schoel Biewigt School 94-7 3*School 10 
Bi; = Y10 
Bo = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 0000 
0.000 
0291 
0.622 
0382 
0.001 
0.001 
0874 
0.087 
0.057 
0.000 
0.952 
08a 
0531 
0.892 

Fixed Effects 
Level Variance Components [Variance __[Sigma”_— |_| 
Student levell T0229 0.238] 


“Duro = Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.18: Motivation to Read: Extrinsic Motivation 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + B3, *Treatment + B,,*Grade 6 + B;;*Grade 6*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model. "5A School Biewigt School 94-7 3*School 10 
Bi, = Yio 
Bo, = 20 
B3) = ¥30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0-000 
School ty, iY 022] =a] 9S 0a] 0.000 
0317 
0-726 
0413 
School 6 ys | 093] =e]. _——0.001 
0.001 
0.893 
0.088 
0.058 
0.000 
0.838 
0270 
0.101 
Freaiment-Grade6 py | 0090063] oa] __1aaxf__0.135 

Fixed Effects 
Level Variance Components [Variance __[Sigma”_— |_| 


Student 0.230 0.237] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.18: Motivation to Read: Extrinsic Motivation 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dyypo + B3, *Treatment + By,*Black + B;;*Black*Treatment + 1; 


Bo; = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model. "5A School Biewigr School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0.000 
0.063 
0.197 
0.523 
0.456 
0.020 
0320 
0.822 
0.092 
0575 
0.000 
0.984 
0726 
0.000 
0735 

Fixed Effects 
Level Variance Components [Variance Sigma” |_| 
Student levell T0253] 0.2381 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.18: Motivation to Read: Extrinsic Motivation 
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E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest,; = Bo; + B,;*Pretest + B.,*Dypo + B3; *Treatment + By; 


*Pretest*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model." 5-# School Biewggr School 94-7 g*School 10 
Bi, = Yio 
Bo = 20 
B3) = 30 
Baj = Y40 

Fixed Effects 

Level 

Student 0000 
0.000 
0302 
0.692 
0412 
0.001 
0.001 
0.853 
0.050 
0.059 
0.000 
0.925 
0.882 
0.406 
ae ears, (Pe |e] 

Fixed Effects 
Level Variance Components [Variance [Sigma” {|_| 
Student levell T0230 0.238} 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.19: Motivation to Read: Social Motivation 


I. Treatment Impact Results 
Level 1 Model Posttest; = Bo; + B,;*Pretest + By;*Dypro + B3, *Treatment + 1;; 
Boj = Yoo + Yor*School 1 + Yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 
Yo7* School 8+ yYog*School 9+ yo9*School 10 


Level 2 Model 


Bi; = Y10 
Boj = Y20 
Bsi = Y30 
Fixed Effects 

Level 

Student 0-000 
0.001 
0.359 
0815 
0.635 
0.068 
0.07 
0.225 
0241 
0.027 
0.000 
eaten 058 asi’ se 3 0306 

a Effects 
Level iverianes Component I Varies 106 = Sige fd 
Student leveld E252] 0.035] 0.235] 


“Duro = Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.19: Motivation to Read: Social Motivation 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest; = Bo; + B,,;*Pretest + By;*Duro + B3; *Treatment + B,;*Poverty + B;*Poverty*Treatment + 1; 


Bo; = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model: « -*School' St Ysch School) vestSehoal 10 
Bi = Y10 
Boj = Y20 
B3; = Y30 
Baj = Y4o0 
Bsi = Ys0 

Fixed Effects 

Level peffeet___f Impact (6) | Se} oF 5} 

Student 0-000 
Schoo yg ana 903 si] 0.003 
0330 
0834 
0.689 
School 6. vos «| TT 006] ent 8s2[ 00a 
0.101 
0.216 
0.196 
0.024 
0.000 
0232 
0.087 
are 5 963 a] 0 

Fixed eitects 
Level Variance Components [Variance __|Sigma”_— | | 
Student levell T0273} 0.234 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.19: Motivation to Read: Social Motivation 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest;, = Bo; + B,,;*Pretest + By;*Dyro + B3; *Treatment + By,*Male + B;*Male*Treatment + 1; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pevel 2 Model: -, -SSchool' St Yeh School) vestSchoal 10 
Bi = Y10 
Boj = Y20 
B3; = Y30 
Baj = Y4o0 
Bsi = Ys0 

Fixed Effects 

Level ptieet dt Inpaet (@) SE t+ 

Student 0-000 
0.001 
0349 
0.859 
067 
0.07 
0.079 
0.236 
0.260 
0.028 
0.000 
0.246 
0m 0.397 
Dos] 0.062 0.615 

Fixed eitects 
Level Variance Components __ [Variance __|Sigma”_— | | 
Student levell T0252} 0.234 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.19: Motivation to Read: Social Motivation 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest; = Bo; + B,,;*Pretest + By;*Dyro + B3; *Treatment + B4,*Grade 6 + B;;**Grade 6*Treatment + 1; 


Bo; = Yoo + Yor*School 1 + Yo2*School 3+ ¥93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel 2 Model: | -*School' St Ysch School 9+ vestSehoal 10 
Bi = Y10 
Boj = Y20 
B3; = Y30 
Baj = Y4o0 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0-000 
School Ly 092s 90-3208] 0.001 
School 3. tus +0057] ~~ —SCSC S| CSC« 
068 
0682 
School 6 tus + ato] onsf 921770077 
0.079 
School. Yo + 0.09007] e126 ___ 0207 
0.283 
0.029 
0.08 0.000 
0.054 0.225 
0.094 0.034 
0.062 0.559 

Fixed Effects 
Level Variance Components [Variance __|Sigma”_— | | 
Student levell 25S} 234 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.19: Motivation to Read: Social Motivation 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest; = Bo; + B,,;*Pretest + By;*Dyro + B3; *Treatment + B,;*Black + B;,*Black*Treatment + r; 


Bo; = Yoo + Yor*School 1 + yo2*School 3+ Y¥93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pevel 2 Model: | -*School' St yeh SChOok9-+ vestSehoal 10 
Bi = Y10 
Boj = Y20 
B3; = Y30 
Baj = Y4o0 
Bsi = Ys0 

Fixed Effects 

Level ptieet Impact FS ot 

Student 0-000 
0.131 
0231 
0.965 
0591 
0330 
07 
0.250 
0.362 
0.359 
0.000 
0.895 
o i a 0.000 
0.049 0.460 

Fixed eiects 
Level Variance Components [Variance __|Sigma”_— | | 
Student levell 273) 0.228] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest; = By; + B,,;*Pretest + By;*Dypro + 63, *Treatment + B,,*Pretest*Treatment + 1; 


Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥o3*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Fevet a ModE: eSchool' Ss yeh SchOok9-+ vestSehool 10 
Bi, = Yio 
Bo = 20 
B3; = 30 
Baj = Y4o0 
Fixed Effects 
Level oe ee ee 
Studeni 
es eG 
Fixed attests 
Level Variance Components [Variance __|Sigma”_— | | 
Student level) TSS] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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0.000 
0.001 
0.357 
0.838 
0.640 
0.066 
0.069 
0.218 
0.230 
0.026 
0.000 
0.215 
0.250 
0.684 
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Table A.20: Motivation to Read: Efficacy 


I. Treatment Impact Results 
Level 1 Model Posttest, = Bo; + B,;*Pretest + By;*Dypo + B3, *Treatment + 1;; 
Bo; = Yoo + Yor*School 1 + yo2*School 3+ ¥93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Panel? Model. yet school Says? School Se yastSehool 10 
Bi; = Y10 
Bo, = 20 
B3i = Y30 

Fixed Effects 

Level 

Student 0.000 
0.000 
0314 
0.467 
0958 
0.008 
0.003 
0577 
0015 
0.004 
0.000 
0.193 
018 

Fixed Effects 
Level Variance Components [Variance ficc_____|sigma” | 
Student level! CT 283] 0.066] 0.209] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.20: Motivation to Read: Efficacy 


A: Poverty*Treatment Interaction Model 


Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + B3, *Treatment + B,,*Poverty + B;;*Poverty*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ yo3*School 4+ yo4*School 5+ yos*School 6+ Yo¢*School 7+ 


Pavel? Model." 5_ A School Biewigr School 94-7 3*School 10 
Bi; = Y10 
Bo = 20 
B3) = ¥30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.000 
0301 
0.463 
0.964 
0.008 
0.004 
0.587 
0013 
0.004 
0.000 
0.186 
0.290 
0.292 
0573 

Fixed Effects 
Level Variance Components [Variance ___[Sigma”_— |_| 


Student 0.288 0209p CT 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.20: Motivation to Read: Efficacy 


B: Gender*Treatment Interaction Model 
Level 1 Model Posttest, = B,; + B,;*Pretest + B2,*Dypo + B3, *Treatment + By,*Male + B;°*Male*Treatment + 1; 
Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model." 5_ A Schoel Biewigt School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 0000 
0.000 
031 
0.469 
0.952 
0.008 
0.003 
057 
0.013 
0.004 
0.000 
0.197 
0225 
0-746 
0-750 

Fixed Effects 
Level Variance Components [Variance __[Sigma”__— |_| 


Student 0.282 o2io) 


“Duro = Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Table A.20: Motivation to Read: Efficacy 


C: Grade*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dypo + B3, *Treatment + B,,*Grade 6 + B;;*Grade 6*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model. "5A School Biewigt School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 
Fixed Effects 

Level 

Student 0000 
0.000 
0304 
0.664 
0988 
0011 
0.004 
0.607 
0.021 
0.005 
0.000 
0.176 
0.059 
0.006 
0217 

Fixed Effects 
Level Variance Components [Variance [Sigma”_—— {|_| 
Student level) E287 0.208] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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Table A.20: Motivation to Read: Efficacy 


D: Race*Treatment Interaction Model 
Level 1 Model Posttest, = Bo; + B,;*Pretest + B2,*Dyypo + B3, *Treatment + By,*Black + B;;*Black*Treatment + 1; 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Pavel? Model. "5A School Biewigr School 94-7 3*School 10 
Bi; = Y10 
Bo, = 20 
B3; = 30 
Byj = Y40 
Bsi = Ys0 

Fixed Effects 

Level 

Student 0000 
0.003 
0.250 
0.423 
0941 
0.029 
0.089 
0.560 
0.021 
0.035 
0.000 
0.161 
0-890 
0.104 
0305 

Fixed Effects 
Level Variance Components [Variance __[Sigma”_— {|_| 
Student level! E289] 0.208] 


“Duro = Indicator for missing pretest data (0 = non-missing, | = missing). 
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E: Treatment*Pretest Interaction Model 
Level 1 Model Posttest,; = Bo; + B,;*Pretest + B.,*Dypo + B3; *Treatment + By; 


*Pretest*Treatment + 1; 


Level 2 Model 
ES School 8+ Yox*School 9+ Yoo*School 10 


Boj = Yoo + Yor*School 1 + yo2*School 3+ y93*School 4+ Yo4*School 5+ yos*School 6+ yo¢*School 7+ 


Bi, = Yio 
Boj = Y20 
B3; = Y30 
Baj = Y40 
Fixed Effects 

Level 

Student 0000 
0.000 
0313 
0.502 
0.963 
0.008 
0.003 
0.562 
0.019 
0.004 
0.000 
0.192 
oa 
0.455 
fe _ 

Fixed Effects 
Level Variance Components [Variance Sigma” |_| 
Student fleveld E283] 209] 


“Duro = Indicator for missing pretest data (0 = non-missing, 1 = missing). 
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Effect Size Analyses 


In the results section, we illustrated the impact findings in terms of overall treatment effect size 
on the outcome variables. The analyses consisted of two-level models including the 
corresponding pretest measure (grand mean centered), a dummy variable indicating a missing 
pretest value (in non-iLEAP models), the treatment dummy variable evaluating the effect of 
being in the treatment condition, and fixed effects for the school level variable. The effect sizes 
presented in Table 3.8 were computed using the following formula. 


y =F, B 
Treatment /Control __ 1 


Glass'A = 
SDof control group SD of control group 


For iLEAP ELA and Reading subscale impact findings, adjusted means are also presented as 
converted NCEs. The following formula and population statistics (see Table 3.9) were used to 
convert the scores to NCEs. 


y ae lation Mean 
NCEcontrot = 26.06 * - Ss = + 50 
Population SD 


y .— Population Mean 
y Treatment .adj op ey 


NCE 
Population SD 


= 26.06 * 


Treatment 


Table A.21.Population Statistics used to convert to NCEs for iLEAP 


Population Mean Population SD 
iLEAP ELA Overall Score 304.669 71.674 
iLEAP Reading Subscale 305.012 61.424 
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Appendix B: Measures 
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Motivation for Reading Questionnaire for Adolescents 
Spring 2011 


Name: Grade: 


School: Date: 


This survey describes how students feel about reading. Read each sentence below and 
decide whether you agree or disagree with what it says. Circle the number that 
represents your opinion. There are no right or wrong answers. We only want to know 
how you feel. If you do not want to provide an answer, please leave it blank and move 
onto the next statement. 


Strongly Disagree Disagree Agree Strongly Agree 
1 2 3 4 


Based on the Motivation for Reading Questionnaire for Adolescents 
(Cantrell, Almasi, & Rintamaa © 2006) 
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Choose One 


sional | sauce | Aaree | Sfonal 
1. | I visit the library often with friends or family. 1 2 3 4 
2. | like hard, challenging books. 1 2 3 4 
3 i that | will do well reading in my classes next 4 5 3 4 
a If the teacher discusses something interesting | 4 5 3 4 
might read more about it. 
5. | | like it when the questions in books make me think. 1 2 3 4 
6. || read about my hobbies to learn more about them. 1 2 3 4 
7. | lama good reader. 1 2 3 4 
8. | enjoy reading magazines. 1 2 3 4 
9. | loften read to other people. 1 2 3 4 
10. Pee Sua who knows an answer in 4 5 3 A 
11. cae new information about topics that 4 5 3 4 
12. | My friends sometimes tell me | am a good reader. 1 2 3 4 
13. ee more from reading than most students in the 4 5 3 HH 
14. | | like to read about new things. 1 2 3 4 
15. | | like hearing the teacher say | read well. 1 2 3 4 
16. | |sometimes read to my parents. 1 2 3 4 
17. | My friends and | like to trade things to read. 1 2 3 4 
18. ce aad something when the words are 4 5 3 A 
19. | | make pictures in my mind when | read. 1 2 3 4 
20. | | always read exactly as the teacher wants. 1 2 3 4 
21. | | usually learn difficult things by reading. 1 2 3 4 
22. | | don’t like vocabulary questions. 1 2 3 4 
23. | Complicated texts are no fun to read. 1 2 3 4 
24. | |am happy when someone recognizes my reading. 1 2 3 4 
25. | | feel like | make friends with people in good books. 1 2 3 4 
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Choose One 
Strongly : Strongly 
Disagree DIsBaree || Seree Agree 

26. ay parents often tell me what a good job | am doing 1 5 3 4 
with reading. 

57. Finishing every reading assignment is very important 4 5 3 4 
to me. 

28. | | talk to my friends about what | am reading. 1 2 3 4 

39. If lam reading an interesting topic | sometimes lose 4 5 3 4 
track of time. 

30. | | like to get compliments for my reading. 1 2 3 4 

31. | Grades are a good way to see how well you read. 1 2 3 4 

32. | like to help my friends with the reading we do for 4 5 3 4 
school. 

33. | lread to improve my grades. 1 2 3 4 

34. | | enjoy a long, involved story or fiction book. 1 2 3 4 

35. | | like to tell my family about what | am reading. 1 2 3 4 

36. | | try to get more answers right than friends. 1 2 3 4 

37. If the project is interesting, | can read difficult 4 5 3 rt 
material. 

38. enjoy reading books about people in different 4 2 3 4 
countries. 

39. | | enjoy searching for information on the internet. 1 2 3 4 

40. | | always try to finish my reading on time. 1 2 3 4 

At If a book is interesting, | don’t care how hard it is to 4 5 3 4 
read. 

42. | | like to finish my reading before other students. 1 2 3 4 

43. In comparison to my other school work | am best at 4 5 3 4 
reading. 

Te am willing to work hard to read better than my 4 5 3 4 
friends. 

AS. | don’t like it when there are too many new ideas in 4 5 3 4 
the text. 

46. | It is very important to me to be a good reader. 1 2 3 4 

a7. In comparison to other activities | do, it is very 4 5 3 4 
important to me to be a good reader. 

48. | |am avery good reader. 1 2 3 4 


Thank you for taking time to fill out this survey! 
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Classroom Observation Protocol for PRJ 


Teacher: School: Grade: 
Date: Start Time: End Time: Observer: 
Number of Students Present: beg. of class end of class 
Pacing Calendar Expedition and Lesson: Observed Expedition and Lesson: 
FIDELITY OF IMPLEMENTATION 
1. CURRICULUM USE 
1 2 3 4 
Poor Fair Good Excellent 


SCORE 


Teacher rarely refers to the 


Teacher sometimes refers to 


Teacher references the 
curriculum guide most of 


Teacher closely and 
consistently references 


wo F : 3 Fi 
oS curriculum guide or no the guide, but there is no : : , : 
a ae 8 . ‘ ‘8 eh the time and uses it to the curriculum guide and 
rc guide is observable in the indication it is used to : : : 

: organize some lesson uses it to organize lesson 
z classroom. organize lesson content. 
a content. content. 

Notes/Comments: 
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1 
Poor 


2 
Fair 


3 
Good 


4 
Excellent 


SCORE 


Teacher rarely presents 
materials using the 
specific lesson 
components described in 
the curriculum guide. 


Teacher sometimes uses 
specific lesson 
components, but 
presentation is 
inconsistent with the 
design described in the 


Teacher uses specific lesson 


components most of the 
time. Most components 
are used as described in 
the curriculum guide. 


Teacher consistently uses 


specific lesson 
components as suggested 
in the curriculum guide. 
Components are easily 
identifiable. 


guide. ¢ A component may be missing 
or taught out of order. For 
example, the teacher may start 
with the pre-reading activity 
then go back to do Advanced 
Word Study. Or, a teacher 
may run out of time and not get 
to the After Reading Activity. 


OR 

¢ Lesson may contain minor 
elements of outside 
instructional material. For 
example, the teacher may teach 
summarization the way she has 
taught it previously, instead of 
using the methods described by 
PRJ. 


¢ All components for the day are 
¢ Two or more components may present and taught in order. 
be missing or taught out of 


order. 


° Lesson uses only PRJ 
components and not outside 


¢ Lesson contains outside instructional material. 


instructional material, rather 
than relying on PRJ. 


Specific Lesson Components 


(See Training Guide for explanation of components appropriate for specific lesson observed — e.g., advanced word study, before 
reading, reading, after reading, and/or use of PRJ Library, (small group) word study, and assessments). 
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1 2 3 4 
Poor Fair Good Excellent BeORE 
The teacher rarely uses or helps | The teacher sometimes uses The teacher helps students learn | The teacher clearly and 
students understand reading strategies, but not and use reading strategies consistently helps students 
reading strategies. as they were intended. most of the time, but learn and use multiple 
explanations lack clarity. reading strategies. 
¢ Reading strategies may be present 
$ in the lesson, but the teacher does | ¢ Teacher uses at least 1 vocabulary | * At least 1 vocabulary and 1 
- not appear to have a clear or comprehension strategy, but not comprehension strategy are used at 
3 understanding of their purpose. both appropriate times in the lesson 
& OR ¢ Additional strategies may be 
bo ¢ All appropriate strategies are used, presented (appropriate to the 
Ss but clarity is low lesson being taught) 
o ° Teacher makes an attempt to * Teacher uses strategies correctly 
= explain strategies, but these may and clearly 
a be unclear at times. This may be 
5 exemplified by students who are 
confused. 
(See Training Guide for specific reading strategies. This category focuses what was accomplished in class, not what was intended to be 
covered by the curriculum.) 
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1 2 3 4 
Poor Fair Good Excellent Seen 
The teacher rarely provides The teacher sometimes provides | The teacher provides students The teacher consistently 

feedback to students. (e.g., feedback to students. with feedback in a timely provides students with 
allows students to get stuck Feedback may be manner most of the time, immediate feedback (both 
on problem words without inconsistent or delayed. but doesn't always allow corrective feedback in 
providing assistance or students to correct their response to mistakes and 
rarely lets students know ¢ When feedback is given, it is mistakes. reinforcement of correct 
that they are doing well). rarely immediate. responses). Students 

* The teacher misses opportunities ¢ There may be minor delay in receive feedback about 

to correct student errors. providing feedback, such as their initial responses 


correcting a word at the end of a 
paragraph instead of when the 
error occurs. 


followed by further 
opportunities to respond 


* Students may not always be and practice new material 


expected to correctly practice a correctly. 
mistake. An example might be if 
a teacher provides a correction but | * For example, teacher has student 


moves on rather than having the re-read a sentence correctly or 
student repeat the correct word or repeat a mispronounced word 
answer. while reading. 


Feedback/Correction Procedures 


¢ Teacher may use vocabulary and 
comprehension strategies to 
provide additional practice or 
check for mastery (e.g., “I 
say/you say/we say” exercise) 


(See Training Guide for examples of feedback/correction procedures — e.g., help with automaticity and fluency, modeling) 


Notes/Comments: 
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1 2 3 4 
Poor Fair Good Excellent PEORE 
The teacher is rarely The teacher only sometimes The teacher is generally on The teacher is organized and 
organized and not stays on task in guiding task in guiding consistently stays on task 
prepared to lead the instructional activities. instructional activities. in guiding instructional 
class. She is easily distracted or She is focused, organized, activities. She maintains 
seems disorganized. and sets an appropriate a consistently 
pace for instruction most appropriate pace 
¢ Teacher is unfocused or of the time. throughout the class 
unprepared. For example, she period. 
might Be reading ahead in the ¢ The teacher may have a “sense 
teachers guide as stent work | "gf gency” in some aspects of | « The eachor has a sense of 
off topié- during stndent the lesson but not others. Pace urgency” in moving through 
RS A MORE GONE seems varied throughout the lesson components. 
lesson. ‘vation 
connecting student responses -O ation’: tb Organization is apparent by 
g tothe [essen eortent: ete ree apparen fi y 4 having materials prepared and 
a. * General pace during the class aaa abhand: : 
is INCONSISTENT. at hand. ¢ There are NO LAPSES in 
¢ There are MINOR LAPSES in appropriate pace during the 
appropriate pace during the class. 
class. 
Notes/Comments: 
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1 
Poor 


2 
Fair 


3 
Good 


4 
Excellent 


SCORE 


Student Involvement in Learning 


The teacher rarely provides 


opportunities for 
students to express their 
thoughts or ideas. 
Students are expected to 
listen to the teacher and 
most interactions are 


procedural or managerial. 


The teacher sometimes 
provides opportunities for 
students to express their 
thoughts and ideas. The 
teacher asks few 
questions or does not 
integrate student 
responses into the lesson. 


¢ The teacher misses a 
substantial number of 
opportunities to make 
connections. 

¢ The teacher only occasionally 
asks questions or may answer 
questions herself. 

° There is little evidence that the 
teacher listens to or 
acknowledges student 
responses. 


The teacher asks questions 
and listens to student 
responses most of the 
time and often connects 
student ideas to the 
lesson topic or to the 
ideas of other students. 


¢ The teacher misses some 
opportunities to make 
connections. 

¢ The teacher may inconsistently 
listen to or acknowledge 
student responses. 

¢ The teacher may repeatedly 
call on one or two students, 
rather than eliciting responses 
or participation from more 
students. 


The teacher deliberately and 
consistently fosters an 
environment where 
student ideas and 
responses are valued. 
The teacher asks 
questions, listens 
attentively, and tries to 
connect the students’ 
ideas to the lesson topic 
or the ideas of other 
students when possible. 


¢ The teacher has established an 
environment which offers 
opportunities for student to 
read or share their thoughts and 
ideas. 

° The teacher acknowledges 
student responses and uses 
student comments to build 
upon previous and existing 
knowledge. 


Notes/Comments: 
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1 2 3 4 
Poor Fair Good Excellent BeORE 
There is no evidence that the The teacher inconsistently Teacher monitors individual Teacher consistently monitors 
teacher is monitoring monitors student student progress and individual student 
student progress and progress and understanding most of progress and 
understanding during understanding and only the time and responds by understanding during 
class. sometimes adjusts adjusting instruction or class and strategically 
instruction or groupings grouping based on what adjusts instruction or 
in response to what happens in class. grouping assignments as 
happens in class. needed. 
a) ¢ The teacher checks in on some 
Ys students but not all. For ¢ The teacher moves around the 
° 3 example, the teacher may classroom and monitors all 
a6 consistently not monitor students’ progress and/or 
=o students in the back of the understanding. 
3 c= room. Or, the teacher may The teacher listens to students 
a z spend a disproportionate reading or discussing assigned 
o oo amount eae ae nies work, and adjusts grouping or 
5 y BTOUPS Ang DOL NO WATOW. instructional format based 
25 other groups are progressing. student needs. For example, a 
6 3 * The teacher may repeat teacher might recognize that 
26 strategies the same way to each students do not understand a 
group, rather than varying the particular reading strategy, and 
approach based on the needs of has them work on the activity 
the students. together rather than 
individually. Another example 
would be to offer alternative 
strategies or suggestions to 
individual students based on 
what they are struggling with. 
(See Training Guide for explanation of types of student groupings — e.g., whole group, small group, pairs, independent practice) 
Notes/Comments: 
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STUDENT ENGAGEMENT 


5. STUDENT RESPONSIVENESS/ENGAGEMENT 


1 2 3 4 
Poor Fair Good Excellent SEORE 
Students are rarely Students are only sometimes Most students are engaged in | All students are consistently 

responsive when they engaged in reading a variety of reading engaged in various 

are asked to read and do activities and do not stay activities, and most are reading activities and 
g not carry out assigned focused on assigned tasks. able to focus on assigned remain focused on 
8 tasks. tasks. assigned tasks. 
5 ¢ — Significant number (15- 
s 20%) of students are * 1 or 2 students are * — Over the duration of the 
- unfocused and not consistently not class, on average 
= staying on task for the engaged for the bulk students are engaged and 
£ bulk (80%) of class time. (80%) of class time. focused (“All students” 
8 Students who are OR = nearly all students, 
2 disengaged might be * A larger group of nearly all of the time). 
& staring into Spaces students are not ¢ Engaged students are 
3 talking to neighbors, engaged for a shorter reading aloud or 
S ce a ape period of time, following along (looking 
00 oe; ? repeatedly over the at the book), writing in 
< drawing, etc. course of the class student workbook on 
3 period. assignments, 
w participating in 

discussions, etc. 
Notes/Comments: 
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6. STUDENT BEHAVIOR 
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3 - : : SCORE 
Poor Fair Good Excellent 
Students appear to have a very Students appear to understand | Students appear to understand | Students appear to have 
limited understanding of basic behavioral rules and and follow established internalized established 
rules and routines in the routines, but transitions to rules and routines most of classroom rules and routines 
classroom, resulting in a new classroom activities the time. Transitions and follow them consistently. 
significant amount of wasted are rough and result in between various activities Transition from one activity 
class time. wasted class time. are relatively smooth, and to the next goes smoothly, 
little class time is wasted. maximizing good use of class 
oi * Rules and routines are not clearly time for instruction. 
£ followed. There is a significant ¢ At this level, it is clear that rules 
s amount of time wasted by the and routines exist, but students * If students have internalized 
2 majority of students in moving to may need several (1.¢., 3+) routines, then the teacher will only 
- DEP RCHNANES: reminders from the teacher. have to give | or 2 reminders and 
S ¢ Some routines may be new or Students may be slow to respond most students will respond quickly 
a unfamiliar to students, and the or may not quickly begin on the to teacher requests. 
= teacher has to explain routines, next activity. This lack of * Some routines are established by 
2 not just give reminders. internalization of routines leads PRJ—such as reading words aloud 
2 to some wasted class time. in advanced word study, or partner 
u reading in the PRJ Library. Other 
routines will be class and teacher 
specific, but these routines will be 
well known by students and will 
aid the flow of class. For example, 
the class may have a clear routine 
for how to work with a partner or 
how and when to get materials. 
Notes/Comments: 
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; . 3 ‘ SCORE 
Poor Fair Good Excellent 
The teacher does not have There are frequent disruptions Classroom activities remain free | Disruptions in classroom 
control of the classroom and during the lesson, and the of disruptions most of the activities are rare, and when 
students are not engaged in teacher has minimal time, and when they do they do occur, teacher 
any purposeful classroom control over keeping occur, the teacher responds quickly to get class 
activities. students focused. responds to re-establish back on track. Behavior 
Students only sometimes order. Students generally management is not an issue 
stay on task if they are not stay engaged in assigned because teacher anticipates 
directly supervised by the activities, and teacher and prevents problems 
teacher. spends minimal time before they occur. 
- dealing with behavior 
EG ¢ Teacher has to stop lesson several management tasks. * Teacher proactively anticipates 
8 E times to talk to students about problems and heads them off 
3 e behavior a * Teacher has to stop lesson to (rather than reacting after the fact) 
o§s * Teacher does not anticipate deal with disruptive behavior . Teacher consistently redirects a 
= disruptive behaviors only a few times potentially disruptive student by 
° Teacher consistently responds to * Teacher anticipates students’ engaging him or her in classroom 
disruptive behavior by yelling at behavior and redirects attention discussion or activities (¢.g., “I 
students or responding to prevent a disruption (e.g., don’t hear you reading...” or 
inappropriately (e.g., “Stop doing strategically pairing students for “___, could you please read 
that —I told you to stop!”) reading activities) next.”) 
Notes/Comments: 
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PRJ Teacher Interview Protocol — Spring 2011 


Thanks for talking to us again today about your experiences with the Passport Reading Journeys 
supplemental reading curriculum. Just to clarify, | work at SEDL, and am part of an independent 
external research team looking at the use and outcomes of the PRJ curriculum use in schools 
participating in the Striving Readers grant in Louisiana. We are independent from PRJ and LDE and 
so would encourage you to feel comfortable being candid about your experiences. We'll be talking 
about the same sort of issues we covered in the interview last fall. I’d like to tape our conversation 
again, if that’s ok with you. All of your comments here will be kept confidential and anonymous, 
meaning that nothing you say will be connected with your specific name or classes (so — is it ok with 
you to tape our conversation?). 


Date Interviewer 


Teacher Name School 


Background Information 


Highest degree 


Certified in any content area(s)? 


Number of years: teaching? reading classes? in this district? 


Support 


Let’s start with the types of support you have received since August to help you with implementing PRJ 
in your classes: 


1) Did you attend the Cohort Meeting in Alexandria in March? 
> Was it helpful to you? (why or why not...) 


2) Have you used any of the available VPORT online resources to assist you in implementing PRJ? (if 
yes): 
> Which one(s)? 
> Have they been helpful? 


3) What assistance have you received from the PRJ coaches? 
> What contact have you had with them (number of classroom visits? telephone calls? emails? 
other?) 
> What happens ona “typical” visit from a PRJ coach? 


4) What kind(s) of support have you received from administrators at your school (e.g., principal, 
assistant principal, director of curriculum and instruction) 
> Have they visited your classroom to observe PRJ lessons? (How many times and for how long?) 
> Discussed PRJ implementation with you? 
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Have you received any support from your school district representative (offer name of District 
Coordinator) in implementing PRJ? (if yes): 
> What type of assistance have you received? 


Have you been in contact with representatives from the Louisiana Dept. of Education? (if yes): 

> What type(s) of contact have you had (and how often)? (e.g., classroom visit, phone contact, 
emails, etc.) 

> What kind(s) of assistance have you received from them? 


Thinking about all forms of training and support that you have received, what has been the most 
helpful to you in implementing PRJ? Why? 


General Classroom Experience 


Now let’s talk about your general experience with PRJ in the classroom so far this year. 


8) 


9) 


10) 


11) 


12) 


13) 


Do you have adequate materials to provide PRJ instruction? (e.g., guides, student workbooks, access 
to computers, etc.) 


Have you established a pacing calendar for the year? (if yes): 

> How many days do you spend on each Expedition (i.e., any re-teach days? how many and 
where are they placed in the Expedition cycle?) 

> Have you been able to stay within 5 lessons of the recommended pace so far? (if no, why not?) 


Voyager has defined a number of essential elements that should be present in PRJ classrooms. | 
have a short list of some of these elements and would like your comments on them, based on your 
experience with teaching PRJ so far: 

how closely you follow the PRJ Teacher’s Guide script 

different student groupings — what they look like and how well they work 

student behavior issues 

usually able to cover all of the material in a lesson? 

how are re-teach days used? 

do students generally complete work that is assigned? (i.e., in class assignments and SOLO) 


VVVVV V 


How often do you use VPORT? 

> What do you use it for? (e.g., student enrollment data, entering/monitoring student progress 
data, pacing calendar, parent reports, tracking student SOLO work, etc.) 

> Have you received any assistance in learning how to use VPORT? (from whom?) 


(if not discussed in question 11): How do you monitor your students’ progress in PRJ classes? 

> Do you use PRJ benchmark or other assessments? (if yes: how/what for?) 

> (if not already addressed): How do you use these assessments specifically to help students who 
are having trouble? 


How would you describe your students’ general response to PRJ? Do you think it’s helping them? 
(why/why not?) 
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14) How many school days have you missed this year? What happens with your PRJ classes when you 
are gone? 


15) Are there any other comments you’d like to make about your experience with implementing PRJ so 


far? 


Thanks so much for helping us get some information about how PRJ implementation is going. If you 
think of anything else later that you’d like to add, or if you have any questions about the research 
study, please feel free to call or email me (leave card). 
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Discussion Topics for SR School Administrators/Principals 


Spring 2011 
Date: Interviewer: 
School Administrator/Principal: 
Experience 


¢ How long have you been at this school as the principal? 
¢ What is your primary role at school (if not principal)? 


¢ Described your previous experience with Voyager’s PRJ 


Implementation Support Provided 
¢ What is your role in supporting implementation of the PRJ program? How do you 


provide this support? 
¢ How often do you visit the PRJ classroom? (i.e., frequency & length of visits)? 
¢ Describe a typical visit to the classroom. 


¢ What type of feedback have you provided to the PRJ teacher (format, content). Provide 
an example. 


¢ What has been the most challenging aspect of the PRJ program to implement? 


¢ What is your perception of how implementation is going in his/her school? What 
evidence are you using to reach this conclusion? 


Implementation Support from Voyager 
¢ What types of support have been provided to and your teacher by Voyager? 
o How often did you and/or your teacher receive support from Voyager? 


¢ What type of interactions do you have with the Voyager coach either before or after they 
visit the PRJ classroom? 


¢ What types of suggestions has the Voyager coach made for you and/or your teacher to 
increase implementation? 


¢ Did you feel your school was adequately prepared to implement the PRJ program? 


¢ What additional supports from Voyager would be beneficial in implementing the PRJ 
program? 
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Implementation Support from LDOE 
¢ What type of support was provided by your LDOE support? 
o How helpful was the support provided? 
o How often did you receive support from LDOE? 


¢ What type of interactions do you have with the LDOE representative either before or 
after they visit the PRJ classroom? 


¢ How has your district coordinator supported implementation of the PRJ program? 


¢ What additional supports for implementation would you recommend to LDOE? 


Challenges/Barriers 
¢ What are some challenges you have experienced with implementing the PRJ program at your 


school? 


Successes 
¢ What is going well with implementation — what’s working well? 


Participating in Research 
* Given the evaluation was separate from implementing the PRJ program, what are some 


challenges you experienced with being involved in the evaluation (i.e., control group, 
randomization)? 


Anything else you can tell us about the implementation of this program? 
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Discussion Topics for Voyager Coach Phone Discussion 


Experience/Background 
¢ How many years have you been training and providing support for Passport Reading 
Journeys (PRJ)? 
¢ Describe your background prior to joining Voyager/Cambium. 


Perception of Implementation 
¢ How frequently did you visit each of your assigned schools? How did the length and 


number of visits vary between the schools? (specific to each Parish/teacher) 

¢ Describe your typical visit with each teacher. 

¢ What types and amount of TA/contact were you able to provide to teachers beyond the 
in-person visits? 

¢ Based on your teachers, what was the most effective means of communication (i.e., 
email, phone, in-person)? Also, what was the most frequently requested type of 
interaction? 

¢ What was the most frequent request for assistance (i.e., SOLO time, V-Port, Classroom 
Management, Time management)? 

¢ How well was each of your teachers able to implement the PRJ program? (Ask about 
each teacher individually, but also overall) 

* How the SR implementation compares to the ‘typical’ implementation in Louisiana(focus 
on different assistance not classroom implementation)? 

¢ What is the role of the principal and/or school administrator in the implementation at 
each of your schools? 

¢ What additional supports for teachers would you recommend to help teachers implement 
the program? 


Challenges/Barriers 
¢ What challenges are you having with implementing the grant? How are these barriers 


impacting your ability to support the implementation of PRJ? 
¢ What challenges are teachers having (both school by school and overall) with 
implementing PRJ? 


Successes 
¢ What is going well with implementation — what is working well? 


Anything else you can tell us about the implementation of this program? 
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LA SR District Coordinators Survey 


Thank you for taking the time to complete our survey! We are interested in gathering information 
about your role, as district coordinator, in the implementation of the Louisiana Striving Readers (LA 
SR) grant, and the Passport Reading Journeys (PRJ) program. As part of the evaluation, your 
answers will be kept anonymous and reported at an aggregated level (e.g., three of the four district 
coordinators). 


1) What is your title within the district (i.e., assistant superintendent)? 


2) How long have you been working within your district? 


Years 


3) What is your background with PRJ before the LA SR grant? 


QO) New to PR] this year 

QO) Previously used PRJ as an administrator 
Q) Previously used PRJ as a teacher 

QO) Previously used PRJ as a reading coach 


4) What activities did you engage in during the past year for the Louisiana Striving 
Readers grant? 


QO) Attended LA SR project meetings and trainings. 

QO) Coordinated release time for teachers and principals to attend trainings associated with 
the LA SR project and PRJ 

QO Assisted principals and teachers with scheduling. 

QO) Assisted SEDL with evaluation activities. 

Q) Supported the literacy intervention teachers with student assessments and records. 

QO) Provided support and assistance to principals, teachers, and PRJ staff in securing and 
obtaining PRJ curriculum materials. 

O) Developed and maintained communication between the LA SR schools, LDOE, and 
SEDL. 
QO) Prepared for and participated in LA SR project evaluation activities. 
QO) Managed funding allocations for the schools. 

O 
QO) Other (please specify) 


If you selected other, please specify 
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5) Please rank order, by frequency, the following types of requests you received from PRJ 
teachers and principals this past year. 


PRJ materials 

Classroom materials (i.e., computers) 

PRJ implementation (i.e., SOLO and Vport) 
Budget 

Scheduling 

Evaluation activities 

Other (please specify) 


6) What additional supports would have been helpful for your teachers and/or principals 
in implementing this grant and program? 


7) Please share any additional information you think would be helpful for us to know 
about the implementation of this project. 


Thank you for taking the time to complete our survey. We enjoyed working with all of you over the past 
year! Wishing you and your district all the best. 
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Discussion Topics for LDOE Consultant Phone Discussion 


Experience/Background 
¢ What is your background and training related to language and literacy programs? 


¢ How long have you worked for LDOE and in what capacity? 
¢ What is your previous experience with PRJ? 
¢ Describe your background before joining LDOE 


Perception of Implementation 
¢ How frequently were you able to visit the participating Parishes, schools, and teachers? 


¢ Describe your typical visit to an intervention teacher and/or district coordinator. 

¢ What amount of TA/contact were you able to provide to teachers and district 
coordinators? Were the majority of your TA requests from teachers or district 
coordinators? 

¢ What was the most frequent request for assistance or clarification (1.e., SOLO time, V- 
Port, Classroom Management, Time management) you received from teachers and/or 
district coordinators. 

¢ How well do you feel teachers are implementing the PRJ program? What evidence do 
you have to support that perception? 

¢ What role to the principals and other school administrators have in supporting 
implementation of the PRJ program? 

¢ What additional supports would you recommend for the teachers to implement PRJ? 


Challenges/Barriers 
¢ Describe the challenges are you experiencing with monitoring implementation of the 
grant vs. PRJ? 
¢ What challenges teachers are having — both school by school as well as overall teachers — 
with implementing PRJ? 


Successes 
¢ What is going well with implementation, both the grant and PRJ? 


Anything else you can tell us about the implementation of this program? 
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Louisiana Striving Readers Project 
Classroom Visitors Log 


Date: Grade Level: 6 [] 7h [] Number of Students: 
SR Teacher Interventionist: 
District/School: 
Visitor’s Name: Position: 
Organization/ Affiliation: [] LDOE [] Voyager 
SEDL Other 


[| 


Time in/Time out j 


[ 


Purpose of visit: 


Observation | _] Coaching L] Technical Assistance L] Data 
Collection 


L] Other 


VISIT SUMMARY 


Focus of visit 


Discussions with teacher 


Knowledge gained 


Teacher concerns 


Is follow up required? [] YES [ NO 


Follow-up suggestions 
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